Nsys Cortex

Troubleshooting

"Failed to create LLM provider" / no API key nsys-cortex chat needs an API key for the active provider. Set it with the config command (schema-validated, written 0o600):

nsys-cortex config set llm.providers.anthropic.apiKey sk-ant-...
nsys-cortex config get llm.defaultProvider                         # confirm which provider is active

Or export the matching env var (ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY, XAI_API_KEY, MISTRAL_API_KEY) — env always wins over the config file. If the key is set but still failing, check you set it under the active provider (config get llm.defaultProvider); the key is only read for whichever provider is selected.

"OpenAI API error 404 (POST https://api.anthropic.com/chat/completions): (empty body)" (or any provider error naming another provider's host) The request went to the wrong endpoint. The provider factory reads the per-provider <PROVIDER>_BASE_URL (OPENAI_BASE_URL, ANTHROPIC_BASE_URL, …), which the config loader fills from llm.providers.<p>.baseUrl; the shared NSYS_CORTEX_LLM_BASE_URL applies only to the provider selected by NSYS_CORTEX_LLM_PROVIDER. Check both with env | grep BASE_URL and nsys-cortex config get llm.providers. Before this was fixed, the loader wrote the default provider's URL into the shared variable and every provider read it, so NSYS_CORTEX_LLM_PROVIDER=openai or a failover chain naming OpenAI hit Anthropic's host whenever defaultProvider was anthropic.

"Refusing to start in dev mode" The CLI refuses dev mode unless NSYS_CORTEX_ALLOW_INSECURE_DEV=1 is set. Dev mode runs the agent as your user — a prompt-injected agent can read or write any file your user can. Acknowledge the risk by exporting the env var, or use --mode=firecracker for hardware-isolated execution. See Configuration → Security.

Agent bundle not found Run npm run build:agent before creating neurons. The CLI requires the pre-built bundle at packages/cortex-agent/dist/index.js.

Neuron connection timeout The neuron agent retries connection for up to 30 seconds. If it still fails, check:

Permission denied (Firecracker) Ensure /dev/kvm is writable by your user:

sudo usermod -aG kvm $USER
# Log out and back in

install.sh / install.ps1: "refusing to run as root" / "refusing to run elevated" The installers run npm install -g into your own prefix and never need more than your user. Run them as yourself; pass --prefix DIR (-Prefix DIR) to install somewhere you own if npm's global prefix is not writable.

install.sh / install.ps1: "Node.js 24 or newer is required and was not found on PATH" (exit 2) The scripts never install Node — they print the route (nvm, brew install node@24, winget install OpenJS.NodeJS.LTS, nodejs.org). Install it, open a new shell, run the script again.

install.sh: "nsys-cortex --version printed X, expected Y" The copy in the prefix's bin answers with another version — an older install in the same prefix that npm did not replace, or a --version that is not a published release. npm ls -g @nsys-cortex/cli shows what is installed; npm uninstall -g @nsys-cortex/cli and run the script again.

install.sh: "<dir>/bin is not on your PATH" The install succeeded; the shell just cannot find it. Add the directory to PATH (or call the printed path directly). With --prefix DIR that is DIR/bin; without, $(npm prefix -g)/bin.

"Docker sandbox unavailable (…) — falling back to dev mode" The default mode auto probes docker version and docker image inspect <image> at startup; when either fails it runs in dev mode (and then refuses without NSYS_CORTEX_ALLOW_INSECURE_DEV=1). The reason in the parentheses says which: start the Docker daemon (or Docker Desktop), run npm run build:image, or point NSYS_CORTEX_DOCKER_BIN / NSYS_CORTEX_DOCKER_IMAGE at the right CLI and image. To fail closed instead of falling back, pin the mode to docker (nsys-cortex config set mode docker or --mode=docker). If Docker is available but the CLI still runs dev mode, the config file pins it: files created before auto became the default carry "mode": "dev" — run nsys-cortex config set mode auto (the config command needs no acknowledgement).

Docker mode: "Neuron image ... not found" Build it once with npm run build:image (needs Docker; node-pty compiles inside the image so no host toolchain is required). To use a different tag, set NSYS_CORTEX_DOCKER_IMAGE. After changing the agent, rebuild the image — docker mode runs the bundle baked into it, not packages/cortex-agent/dist/.

Docker mode: "Docker daemon not reachable" docker version must work for the user running nsys-cortex. On Linux that usually means membership of the docker group (sudo usermod -aG docker $USER, then re-login) or a running Docker Desktop. A different CLI can be pointed at with NSYS_CORTEX_DOCKER_BIN.

Docker mode: "boot failed: Neuron did not connect within ...ms" The container started but the agent never reached the host socket. The error carries docker's last stderr lines — look there first. Common causes: the host socket dir could not be bind-mounted (on Docker Desktop a Unix socket cannot be shared with the container, which is why auto selects the loopback TCP channel there — NSYS_CORTEX_DOCKER_TRANSPORT; if you forced unix on Desktop, drop that; see Security Posture → Docker mode); NSYS_CORTEX_DEFAULT_MEM_MIB set too low (the agent is OOM-killed on start); a very deep NSYS_CORTEX_OVERLAY_STORE_DIR/home making the socket path exceed sun_path. docker ps -a --filter label=nsys-cortex.managed=true shows what is left.

Docker mode (tcp channel): "the channel port was not published within the boot timeout (…)" The host polls docker port <container> 9999/tcp until the daemon reports the loopback port the agent's channel port is published on. It never answered: the container did not start (the error carries docker's stderr lines), or the daemon reports the port elsewhere. docker port nsys-cortex-<id> 9999/tcp by hand shows what the daemon says; anything but 127.0.0.1:<port> is refused by design (the channel must not be reachable from the LAN).

Docker mode (tcp channel): "neuron closed the channel before answering (wrong token?)" The host dialed the published port until the boot timeout and every connection closed before the agent answered. Docker's port proxy closing the first dial is normal (the agent is not listening yet; logged at info). A boot that fails this way means the agent never accepted the hello: the token file (~/.nsys-cortex/sockets/docker/<id>/token, written fresh per boot, read by the agent at /run/nsys-cortex/token) is not the one the container mounts, something else answers on that host port, or the image's agent predates the tcp channel (npm run build:image).

Docker mode: "the offline network needs the Unix-socket channel (NSYS_CORTEX_DOCKER_TRANSPORT=unix)" NSYS_CORTEX_DOCKER_NETWORK=none has no port to publish. Set NSYS_CORTEX_DOCKER_TRANSPORT=unix (Linux Docker Engine); on Docker Desktop the offline network is not available with the tcp channel (ADR-009).

Docker mode: "boot failed: container exited during boot (code 125, …)" / "docker client failed to start … ENOENT" create() fails the moment the docker client exits, carrying docker's own stderr lines — it does not wait out bootTimeoutMs. Exit code 125 is the daemon refusing the run (image missing or built for another architecture, a mount source that is not a directory, an unknown --network); 126/127 point at the entrypoint inside the image; "failed to start … ENOENT" means NSYS_CORTEX_DOCKER_BIN (default docker) is not on PATH. Nothing is left behind: the state file, the socket dir and the container name are rolled back, so the next create() starts clean.

Docker mode: list shows a neuron as running but exec says "not running" Same rule as dev mode: the control socket is owned by the process that created the container. If that process has exited (the daemon keeps a container whose docker run client was killed), exec/chat/shell restart the neuron in the current process — the orphaned container is reaped by label. If it is still alive, they refuse: Error: Neuron <id> is running under another process (pid N, since …): send commands through that process (the HTTP API server), or run nsys-cortex destroy <id> first — the ownership check reads the state file's client pid and verifies --name <container> on its command line. nsys-cortex destroy <id> works from any process.

Port 3100 in use Set NSYS_CORTEX_PORT=3200 (or any free port) as an environment variable.

nsys-cortex list shows a neuron but commands fail Cross-process discovery makes neurons visible across CLI invocations, but only the process that created the neuron can send commands to it — the vsock Unix socket server is owned by the creating process. If that process is gone, exec, install, shell, or chat with the same agent ID restart the neuron in the current process; if it is still alive (an API server, a chat in another terminal), they refuse with its pid rather than take the neuron away from it — send the command through that process (the HTTP API server, npm run start, is the multi-process path), or nsys-cortex destroy <id> first.

nsys-cortex list shows a neuron as "stopped" A stopped neuron means the overlay directory exists (persistent data is preserved) but no running process was detected. This happens after destroy with --wipe omitted, or after the host process exits. The neuron's files are still available — create with the same agent ID will reuse the overlay and all data will reappear. Use destroy --wipe to permanently delete the overlay. nsys-cortex destroy <id> and destroy <id> --wipe act on a stopped neuron (the first leaves its files, the second removes them), and so does DELETE /neurons/<id>; "does not exist" means neither a neuron nor an overlay directory.

Stale neuron shown in list after crash If the host process crashes without calling destroy(), the neuron is detected via three-tier discovery: .neuron.json state file (PID liveness check), vsock socket file on disk, or lsof for open sockets. Dead processes are automatically cleaned up — stale state files and socket files are removed. If the agent process itself is still alive (orphaned), it will self-terminate when it detects the vsock connection is lost.

Invisible zombie agent process after host crash The .neuron.json state file is written with pending: true BEFORE the boot-time health check; the flag is cleared on success. If the host dies between spawn and health-check (a roughly 15-second window), the next list() finds a stale pending state file older than 60s and SIGKILLs the orphaned agent process. No manual cleanup needed. You can also force a sweep by importing DevCortexSandboxManager and calling cleanupOrphans() — note this uses pgrep -f cortex-agent and may kill agent processes belonging to other concurrent sessions on the same host.

API server returns 401 Unauthorized The HTTP API enforces bearer-token auth on all /api/ routes when NSYS_CORTEX_API_TOKEN is set. Pass Authorization: Bearer <token> on every request. The token is required to bind anything other than loopback; otherwise the server refuses to start.

API server returns generic INTERNAL_ERROR with no detail Error envelopes intentionally don't echo internal messages (provider SDK errors can leak Authorization headers). The full error and stack are logged with the requestId shown in the response. grep the log: grep <requestId> ~/.nsys-cortex/logs/nsys-cortex-YYYY-MM-DD-pid*.log | jq .

Tests timing out Integration tests create real neuron processes and may take up to 30 seconds. Ensure npm run build:agent has been run first. If tests hang, check for orphaned agent processes: pkill -f "cortex-agent".

Viewing log files Structured JSON logs are written to ~/.nsys-cortex/logs/ by default. Files are named nsys-cortex-YYYY-MM-DD-pidNNNN.log (the PID suffix prevents concurrent cortex processes from interleaving), mode 0o600. To tail today's logs across all running processes:

tail -f ~/.nsys-cortex/logs/nsys-cortex-$(date +%Y-%m-%d)-pid*.log | jq .

To find a specific request: grep <requestId> ~/.nsys-cortex/logs/nsys-cortex-*.log | jq .. Known credential patterns (API keys, Bearer tokens, PEM keys, etc.) are automatically redacted to ***REDACTED*** before lines hit disk — but don't rely on this for arbitrary secrets, and audit any custom code that calls logger.info with sensitive payloads.

To disable file logging, set NSYS_CORTEX_FILE_LOGGING=false or "logging": { "fileLogging": false } in config. To disable console logging, set NSYS_CORTEX_CONSOLE_LOGGING=false. Console and file have independent log levels — e.g., set NSYS_CORTEX_CONSOLE_LOG_LEVEL=warn for quiet console while keeping NSYS_CORTEX_FILE_LOG_LEVEL=debug for verbose file logs. To change the log directory, set NSYS_CORTEX_LOG_DIR=/path/to/logs.

shell in Firecracker mode: "node-pty is not available" The PTY shell in Firecracker mode requires node-pty to be installed inside the VM rootfs. Ensure the rootfs image includes python3, make, and gcc (build tools for node-pty native compilation), or pre-install node-pty during rootfs build. In dev mode, node-pty is installed as a regular npm dependency and the shell uses direct PTY passthrough — no extra setup needed.

shell in Firecracker mode: frozen / no output The PTY read loop uses long-polling with a 5s agent-side timeout and 10s vsock timeout. If the vsock connection drops, the read loop exits and the shell closes. Check that the neuron is still running with nsys-cortex list. If the shell freezes, press Ctrl+C or Ctrl+D to force cleanup.

Too many log files Old log files are automatically cleaned up at startup based on the retention period (default 30 days). To adjust, set NSYS_CORTEX_LOG_RETENTION_DAYS or "logging": { "retentionDays": 7 } in config. Log files rotate at 25 MB by default — override with NSYS_CORTEX_LOG_MAX_FILE_SIZE_MB.

Sessions disappear after API server restart Before 0.2.0's persistence wiring, only session metadata was persisted to disk — transcripts were in-memory only, so restarting the API server lost every conversation. As of 0.2.0 the CortexSessionManager writes both metadata and transcripts through FileSessionStorage, and the server calls await sessionManager.initialize() before accepting traffic. If you still see sessions vanish after a restart, check ~/.nsys-cortex/sessions/sessions.json exists, the per-agent subdirectories under it (<storageDir>/<agentId>/<sessionId>.jsonl) are readable, and that NSYS_CORTEX_SESSION_MAX_AGE hasn't expired the session you're looking for (default 24h since createdAt).

HTTP API returns 429 RATE_LIMITED All /api/ routes are rate-limited per client IP (600 / 15 min by default), with a tighter budget on the agent-loop routes /sessions/:id/message and /run (60 / 15 min — each can spend real LLM tokens). Heavy scripted use or a load test will trip these. Raise the budgets with NSYS_CORTEX_RATE_LIMIT / NSYS_CORTEX_RATE_LIMIT_LLM, widen the window with NSYS_CORTEX_RATE_LIMIT_WINDOW_MS, or set a limit to 0 to disable that tier. If every client trips the limit at once, you're likely behind a reverse proxy without trust proxy set — all requests share the proxy's IP bucket; configure Express trust proxy so per-client IPs are seen. Check the RateLimit-Remaining response header to see how much budget is left.

CLI chat doesn't remember previous conversations The CLI persists transcripts too (same FileSessionStorage as the API), but each nsys-cortex chat invocation starts a fresh session by default. To pick up where you left off, use nsys-cortex chat <agent> --continue (most recent session for that agent) or --session <id> (a specific one — run /sessions inside a chat to list ids). If --continue starts fresh anyway, the prior session likely expired past NSYS_CORTEX_SESSION_MAX_AGE, or NSYS_CORTEX_SESSION_STORAGE_DIR points somewhere different than when it was written.

Resumed agent forgets which files it read / commands it ran Persistence is faithful as of 0.2.0 — assistant tool calls and their results are stored and reconstructed on reload. If a resumed session still looks prose-only, it was likely created before this fix (its transcript only has role + content text); start a new session to get full fidelity. Extended-thinking blocks are stored but intentionally not replayed to the model on resume (a stale reasoning signature can be rejected), so the agent won't "see" its prior thinking — only its prior actions.

Old sessions appear as idle or vanish entirely Sessions idle past NSYS_CORTEX_SESSION_IDLE_TIMEOUT (default 4h) flip to idle status on the next read — sending a new message via POST /sessions/:sessionId/message flips them back to active. Sessions older than NSYS_CORTEX_SESSION_MAX_AGE (default 24h since createdAt) are dropped: getSession returns undefined and the on-disk transcript is deleted. To extend either, set the env vars before starting the API server.

Chat session randomly dies mid-conversation with a 5xx error The agent loop now retries 408/425/429/500/502/503/504/529 + Node net errors up to 4 attempts with exponential backoff + jitter, honoring Retry-After. A single transient 529 from Anthropic at peak hours no longer kills a 20-turn chat. If retries are exhausted, the run yields an error event followed by run_complete (the loop is wrapped in try/finally so callers always see completion). Check ~/.nsys-cortex/logs/ for "Provider stream retry" warnings.

tool_use ids must be followed by a tool_result provider error This used to fire when long sessions compacted between a tool_use and its tool_result. The compactor now uses snapTailBoundary() to move the cut outward so pairs stay together; assertToolPairsBalanced() asserts the invariant before every provider call. If this error still appears, file a bug — the assertion should fire first with a clear "duplicate tool_use" / "orphan tool_result" message and stop the run cleanly.

Ctrl-C in chat doesn't immediately cancel LLMProviderOptions.signal propagates end-to-end into fetch()/client.messages.stream() and into tool execute(). Ctrl-C aborts the controller and the provider stream closes. Tool execution (e.g. a long bash subprocess) receives the same signal. If you see a Ctrl-C lag, it's likely the tool isn't checking signal.aborted — file a bug against that tool.

chat shows no cache savings even after multiple turns Cache savings only kick in for Anthropic (which uses cache_control breakpoints) and OpenAI (which auto-caches stable prefixes ≥1024 tokens keyed by prompt_cache_key). Verify: