colab.ipynb
Rama main · réplica de lectura del repositorio original.
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "W3WO6QudJfDl"
},
"source": [
"# 🌸 Meet Jun on a free Colab GPU\n",
"\n",
"This is **Jun OS**, a fan-made app where you talk to Jun, the robot girlfriend from *!Ω Factorial Omega*. She talks back with a face that moves and, if you want, a voice. Normally she runs on your own machine. This notebook borrows Google's hardware instead, a bit slower but nothing to install.\n",
"\n",
"> 🔞 **Built on an adult (18+) game.** Consenting adults only.\n",
"\n",
"### What to do\n",
"\n",
"1. **Pick how you'll use her**, in Step 1's `Mode` box:\n",
" - **Everything here** (the default): you chat with her in this browser tab. Colab wipes everything when the session ends, chats and account included.\n",
" - **Inference only**: you chat in the Jun OS app on your PC or phone, where your chats stay. This notebook only does her thinking. *Started this from the phone app's **Start Colab session**? It's picked for you, don't touch anything.*\n",
"2. **Runtime → Change runtime type → T4 GPU → Save.** If Colab offers a **TPU** (v6e-1 or v5e-1), take it, she answers much faster. *(\"Connect T4\" up top means the T4 is already picked.)*\n",
"3. **Runtime → Run all.** If Colab warns that the notebook wasn't written by Google, click **Run anyway**.\n",
"4. **Wait for the ✅ under each step.** The first run downloads her brain: about 10 minutes on a T4, 20-30 on a TPU.\n",
"5. **Step 3 gives you a link** (Everything here) **or a code for your app** (Inference only).\n",
"6. **Leave this tab open.** Step 3 keeps running on purpose. Close the tab and Colab soon decides you're gone and ends the session.\n",
"\n",
"### Privacy\n",
"\n",
"In inference-only mode your messages go from your PC or phone to this notebook through a Cloudflare tunnel (HTTPS, locked with a key that's new every run). Google and Cloudflare are in the path either way. Inference only just keeps your saved chats off their machines.\n",
"\n",
"### GPU or TPU?\n",
"\n",
"- **T4 GPU**: free, always there, runs the same builds as a PC.\n",
"- **TPU**: the fast one. A **v6e-1** runs the full 12B, a **v5e-1** a slightly compressed one, and `TPU_Model` on `auto` picks for you. Kaggle's **v5e-8** has room for a lot more. The first start spends 10-20 minutes preparing the model before she answers. No karaoke on a TPU."
]
},
{
"cell_type": "code",
"metadata": {
"id": "7XRoA3OkJfDn",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "3531efcb-feb4-4831-ccbc-88a8664488c8"
},
"source": [
"#@title ▶️ Step 1 - Get the place set up (~1 min, ~5 on a TPU)\n",
"Mode = \"Everything here\" #@param [\"Everything here\", \"Inference only (Jun runs on my PC or phone)\"]\n",
"#@markdown **Everything here**: chat with her in this tab. **Inference only**: chat in the Jun OS app on your PC or phone, this notebook just does her thinking. Started from the phone app? Leave this alone, the app switches it for you.\n",
"import glob, hashlib, json, os, subprocess, tempfile, time, urllib.request\n",
"\n",
"KAGGLE = bool(os.environ.get(\"KAGGLE_KERNEL_RUN_TYPE\"))\n",
"BASE = \"/kaggle/working\" if KAGGLE else \"/content\"\n",
"REPO_DIR = f\"{BASE}/Jun\"\n",
"repo_name = \"Jun\"\n",
"\n",
"# kaggle can't hand the browser a link into the notebook like\n",
"# colab's proxyPort does, so the whole app can't live here. the\n",
"# brain can.\n",
"INFERENCE_ONLY = KAGGLE or Mode.startswith(\"Inference\")\n",
"if KAGGLE and not Mode.startswith(\"Inference\"):\n",
" print(\"Kaggle can't open a page from inside the notebook, so this run is inference only:\")\n",
" print(\"you chat in the Jun OS app on your PC or phone, this machine does her thinking. Step 3 says what to paste or scan.\")\n",
"\n",
"# the phone app's Colab page puts window.junos in our output frame,\n",
"# Step 3 hands it the pairing link so nobody has to scan anything\n",
"PHONE = False\n",
"if not KAGGLE:\n",
" try:\n",
" from google.colab import output as colab_output\n",
" PHONE = bool(colab_output.eval_js(\"!!(window.junos && window.junos.postMessage)\", timeout_sec=10))\n",
" except Exception:\n",
" pass\n",
"if PHONE:\n",
" INFERENCE_ONLY = True\n",
" print(\"📱 Started from the Jun OS app, so this run is inference only. She connects by herself at Step 3.\")\n",
"\n",
"# missing is fine, the caller falls back. there but unreadable is\n",
"# NOT, colab keeps every secret locked per notebook until you flip\n",
"# its \"Notebook access\" switch, and swallowing that sends you down\n",
"# the public-clone path with a lie about your PAT\n",
"def secret(name):\n",
" if KAGGLE:\n",
" try:\n",
" from kaggle_secrets import UserSecretsClient\n",
" return UserSecretsClient().get_secret(name)\n",
" except Exception:\n",
" return None\n",
" from google.colab import userdata\n",
" try:\n",
" return userdata.get(name)\n",
" except userdata.SecretNotFoundError:\n",
" return None\n",
" except Exception as e:\n",
" print(f\"⚠️ {name} is in Secrets but this notebook can't read it ({type(e).__name__}).\")\n",
" print(\" Key icon on the left > switch on Notebook access next to it > run this cell again.\")\n",
" return None\n",
"\n",
"# colab only shows a child's output when it ran through `!`, so a\n",
"# plain subprocess that fails dies with an exit code and nothing\n",
"# else. keep the output and print the end of it when that happens\n",
"def sh(cmd):\n",
" r = subprocess.run(cmd, shell=True, capture_output=True, text=True)\n",
" if r.returncode:\n",
" print((r.stdout + r.stderr)[-3000:])\n",
" raise RuntimeError(f\"`{cmd}` failed with exit code {r.returncode}, its output is above\")\n",
"\n",
"# v5e and newer show up as vfio groups, v4 and older as accel nodes\n",
"TPU_DEVICES = len(glob.glob(\"/dev/vfio/[0-9]*\")) or len(glob.glob(\"/dev/accel[0-9]*\"))\n",
"gpu = subprocess.run([\"nvidia-smi\", \"-L\"], capture_output=True, text=True).stdout.strip() if not TPU_DEVICES else \"\"\n",
"ACCEL = \"tpu\" if TPU_DEVICES else (\"gpu\" if gpu else \"cpu\")\n",
"if ACCEL == \"tpu\":\n",
" print(\"TPU found, she'll run on vLLM.\")\n",
"else:\n",
" print(\"GPU:\", gpu if gpu else \"none! Runtime > Change runtime type > T4 GPU (or a TPU), Save, then Run all again\")\n",
"\n",
"# a second run in the same session used to skip all of this and\n",
"# keep the first clone forever, so anything pushed in between never\n",
"# showed up. now it pulls, with the same credentials\n",
"updating = os.path.isdir(REPO_DIR)\n",
"clone_env = os.environ.copy()\n",
"# no prompt to answer in a notebook, fail on the spot instead\n",
"clone_env[\"GIT_TERMINAL_PROMPT\"] = \"0\"\n",
"askpass_path = None\n",
"github_pat = secret(\"GITHUB_PAT\")\n",
"github_username = secret(\"GITHUB_USERNAME\")\n",
"if github_pat and github_username:\n",
" print(\"Credentials loaded securely. \" + (\"Updating the copy from the last run...\" if updating else \"Attempting to clone private repo...\"))\n",
" clone_url = f\"https://github.com/{github_username}/{repo_name}.git\"\n",
" fd, askpass_path = tempfile.mkstemp(prefix=\"git-askpass-\")\n",
" script = (\n",
" \"#!/usr/bin/env python3\\nimport sys\\n\"\n",
" f\"username = {github_username!r}\\npassword = {github_pat!r}\\n\"\n",
" \"print(username if 'Username' in sys.argv[1] else password)\\n\"\n",
" )\n",
" os.write(fd, script.encode())\n",
" os.close(fd)\n",
" os.chmod(askpass_path, 0o700)\n",
" clone_env.update(GIT_ASKPASS=askpass_path, GIT_ASKPASS_REQUIRE=\"force\")\n",
"else:\n",
" if not updating:\n",
" print(\"Downloading Jun OS...\")\n",
" clone_url = \"https://github.com/efficiencyx/JunOS.git\"\n",
"\n",
"try:\n",
" if updating:\n",
" result = subprocess.run([\"git\", \"-C\", REPO_DIR, \"pull\", \"-q\", \"--ff-only\"], env=clone_env)\n",
" else:\n",
" result = subprocess.run([\"git\", \"clone\", \"-q\", \"--depth\", \"1\", clone_url, REPO_DIR], env=clone_env)\n",
"finally:\n",
" if askpass_path:\n",
" os.remove(askpass_path)\n",
"if updating and result.returncode != 0:\n",
" print(\"⚠️ Couldn't update the copy from the last run, carrying on with it as it is.\")\n",
"elif result.returncode != 0 and not askpass_path:\n",
" raise RuntimeError(f\"Couldn't clone {clone_url} without credentials. If that repo is private, \"\n",
" \"add GITHUB_PAT and GITHUB_USERNAME to Secrets with Notebook access on.\")\n",
"elif result.returncode != 0:\n",
" raise RuntimeError(\"Clone failed. Check your PAT, username, and repository name.\")\n",
"print(\"Jun OS at\", subprocess.run([\"git\", \"-C\", REPO_DIR, \"log\", \"-1\", \"--format=%h %cs\"],\n",
" capture_output=True, text=True).stdout.strip())\n",
"\n",
"%cd {REPO_DIR}\n",
"\n",
"# inference only still wants php-cli, it draws the phone app's QR code\n",
"print(\"Installing PHP...\")\n",
"sh(\"apt-get -qq update && apt-get -qq install -y php-cli php-mbstring php-sqlite3 php-curl > /dev/null\")\n",
"\n",
"if ACCEL == \"tpu\":\n",
" # its own venv ON PURPOSE. vllm-tpu pins its own jax and torch,\n",
" # and pip-installing that over colab's breaks the notebook kernel\n",
" # in ways a restart doesn't fix. 0.32.0 was two days old when\n",
" # pinned, everything older carries PYSEC-2026-4319.\n",
" print(\"Installing vLLM for TPU (this is the slow bit)...\")\n",
" VLLM_ENV = f\"{BASE}/vllm-env\"\n",
" sh(\"pip -q install uv\")\n",
" # --allow-existing or a second run of this cell in the same\n",
" # session dies on the venv the first one made. and the install\n",
" # below is then a no-op instead of another 5 minutes\n",
" sh(f\"uv venv -q --allow-existing --python 3.12 {VLLM_ENV}\")\n",
" sh(f\"uv pip install -q --python {VLLM_ENV}/bin/python vllm-tpu==0.32.0\")\n",
" # memory_stats can come back empty on some runtimes, the table\n",
" # is per chip and only there for that case\n",
" probe = subprocess.run([f\"{VLLM_ENV}/bin/python\", \"-c\", (\n",
" \"import jax, json; d = jax.devices(); \"\n",
" \"print(json.dumps({'n': len(d), 'kind': d[0].device_kind, \"\n",
" \"'hbm': sum((x.memory_stats() or {}).get('bytes_limit', 0) for x in d)}))\"\n",
" )], capture_output=True, text=True)\n",
" try:\n",
" TPU = json.loads(probe.stdout.strip().splitlines()[-1])\n",
" except (IndexError, ValueError):\n",
" raise RuntimeError(\"vLLM installed but can't see the TPU:\\n\" + probe.stderr[-2000:])\n",
" if not TPU[\"hbm\"]:\n",
" per_chip = {\"TPU v6 lite\": 32, \"TPU v5 lite\": 16, \"TPU v5\": 95, \"TPU v4\": 32, \"TPU v3\": 16}\n",
" TPU[\"hbm\"] = per_chip.get(TPU[\"kind\"], 16) * TPU[\"n\"] * 1024**3\n",
" print(f\"\\n✅ All set: {TPU['n']}x {TPU['kind']}, {TPU['hbm'] / 1024**3:.0f} GB of TPU memory. On to Step 2.\")\n",
"else:\n",
" print(\"Installing Ollama (this is the slow bit)...\")\n",
" sh(\"apt-get -qq install -y zstd > /dev/null\")\n",
" OLLAMA_VERSION = \"0.40.10\"\n",
" OLLAMA_INSTALL_SHA256 = \"25f64b810b947145095956533e1bdf56eacea2673c55a7e586be4515fc882c9f\"\n",
" installer = \"/tmp/ollama-install.sh\"\n",
" urllib.request.urlretrieve(\n",
" f\"https://github.com/ollama/ollama/releases/download/v{OLLAMA_VERSION}/install.sh\",\n",
" installer,\n",
" )\n",
" with open(installer, \"rb\") as stream:\n",
" actual_sha256 = hashlib.sha256(stream.read()).hexdigest()\n",
" if actual_sha256 != OLLAMA_INSTALL_SHA256:\n",
" raise RuntimeError(\"Ollama installer checksum mismatch\")\n",
" install_env = os.environ.copy()\n",
" install_env[\"OLLAMA_VERSION\"] = OLLAMA_VERSION\n",
" subprocess.run([\"sh\", installer], env=install_env, check=True, stdout=subprocess.DEVNULL)\n",
" os.remove(installer)\n",
"\n",
" env = os.environ.copy()\n",
" # loopback only, even in inference-only mode. the tunnel talks to\n",
" # the gate in Step 3 and the gate talks to this\n",
" env[\"OLLAMA_HOST\"] = \"127.0.0.1:11434\"\n",
" env[\"OLLAMA_FLASH_ATTENTION\"] = \"1\"\n",
" env[\"OLLAMA_KV_CACHE_TYPE\"] = \"q8_0\"\n",
" # her and the little title model side by side, one request at a\n",
" # time. same as the PC setup, otherwise naming a chat evicts her\n",
" env[\"OLLAMA_MAX_LOADED_MODELS\"] = \"2\"\n",
" env[\"OLLAMA_NUM_PARALLEL\"] = \"1\"\n",
" subprocess.Popen([\"ollama\", \"serve\"], env=env, start_new_session=True,\n",
" stdout=open(f\"{BASE}/ollama.log\", \"w\"), stderr=subprocess.STDOUT)\n",
" for _ in range(60):\n",
" try:\n",
" urllib.request.urlopen(\"http://127.0.0.1:11434/api/tags\", timeout=2)\n",
" print(\"\\n✅ All set. On to Step 2.\")\n",
" break\n",
" except Exception:\n",
" time.sleep(1)\n",
" else:\n",
" print(f\"\\n⚠️ The AI engine didn't come up. Peek at {BASE}/ollama.log\")"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "qQRL8w4wJfDn",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "ebeb446c-55d8-486d-829d-73cbecd8a81c"
},
"source": [
"#@title ▶️ Step 2 - Give her a brain (a few GB, ~3-6 min, longer on a TPU)\n",
"Model = \"auto\" #@param [\"auto\", \"E2B (smaller, faster)\", \"12B (bigger, smarter)\"]\n",
"# @markdown **GPU:** `auto` grabs the 12B fine-tune - Colab's T4 can handle it, and it's the sharper, more in-character Jun. Pick E2B if you'd rather have snappier replies.\n",
"Quant = \"auto\" #@param [\"auto\", \"Q4_K_M\", \"Q6_K\", \"Q8_0\"]\n",
"# @markdown **Quant:** how much precision she keeps. `auto` takes what the installer would pick for this GPU (Q4_K_M for the 12B on a T4). Higher is sharper but heavier, and leaves less room for context.\n",
"TPU_Model = \"auto\" #@param [\"auto\", \"12B (full precision, v6e-1 and up)\", \"12B INT8 (fits a v5e-1)\"] {allow-input: true}\n",
"Context = \"auto\" #@param [\"auto\", \"8192\", \"16384\", \"32768\"]\n",
"# @markdown **TPU:** `auto` takes the full-precision 12B when the TPU has room for it and the INT8 build on a 16 GB v5e-1. You can also type any Hugging Face repo with safetensors weights (a private one needs `HF_TOKEN` in Secrets). `auto` context takes what fits next to the weights.\n",
"import subprocess, time, urllib.request\n",
"\n",
"JUN_E2B = \"hf.co/efficiencyx/Jun-LoRA-E2B-GGUF\"\n",
"JUN_12B = \"hf.co/efficiencyx/Jun-LoRA-12B-GGUF\"\n",
"TITLE_MODEL = \"hf.co/efficiencyx/Titlewen-GGUF:F16\"\n",
"CTX_TIERS = [6144, 8192, 12288, 16384]\n",
"TPU_REPOS = {\n",
" \"12B (full precision, v6e-1 and up)\": \"efficiencyx/Jun-LoRA-12B-Safetensor\",\n",
" \"12B INT8 (fits a v5e-1)\": \"efficiencyx/Jun-LoRA-12B-Safetensor-INT8\",\n",
"}\n",
"\n",
"def vram_mb():\n",
" try:\n",
" out = subprocess.check_output(\n",
" [\"nvidia-smi\", \"--query-gpu=memory.total\", \"--format=csv,noheader,nounits\"])\n",
" return int(out.decode().splitlines()[0])\n",
" except Exception:\n",
" return 0\n",
"\n",
"# same thresholds as quant_for in install.sh\n",
"def quant_for(model, mb):\n",
" if \"12B\" in model:\n",
" return \"Q8_0\" if mb >= 23500 else \"Q6_K\" if mb >= 15500 else \"Q4_K_M\"\n",
" return \"Q8_0\" if mb >= 7500 else \"Q6_K\" if mb >= 5500 else \"Q4_K_M\"\n",
"\n",
"def model_mb(name):\n",
" tags = json.load(urllib.request.urlopen(\"http://127.0.0.1:11434/api/tags\", timeout=10))\n",
" for m in tags.get(\"models\", []):\n",
" if m.get(\"name\") == name:\n",
" return m.get(\"size\", 0) // 1048576\n",
" return 0\n",
"\n",
"if ACCEL != \"tpu\":\n",
" if Model.startswith(\"E2B\"):\n",
" MODEL = JUN_E2B\n",
" elif Model.startswith(\"12B\"):\n",
" MODEL = JUN_12B\n",
" else:\n",
" MODEL = JUN_12B if vram_mb() >= 12000 else JUN_E2B\n",
" MODEL += \":\" + (Quant if Quant != \"auto\" else quant_for(MODEL, vram_mb()))\n",
"\n",
" print(f\"Downloading {MODEL} - this is the long wait, go grab a drink ...\")\n",
" sh(f\"ollama pull {MODEL}\")\n",
" sh(f\"ollama pull {TITLE_MODEL}\")\n",
" # same sizing as default_num_ctx() in api/lib/providers/context.php:\n",
" # 2 GB of working room, ~0.2 MB of KV cache per token at q8_0. php\n",
" # can't measure this card from your PC, so Step 3 hands it over\n",
" headroom = vram_mb() - 2048 - model_mb(MODEL)\n",
" NUM_CTX = CTX_TIERS[0]\n",
" for tier in CTX_TIERS:\n",
" if headroom / 0.2 >= tier:\n",
" NUM_CTX = tier\n",
"else:\n",
" pick = TPU_Model.strip()\n",
" if pick == \"auto\":\n",
" # 22 GB of bf16 weights, a 16 GB v5e-1 needs the int8 build\n",
" pick = \"12B (full precision, v6e-1 and up)\" if TPU[\"hbm\"] >= 24 * 1024**3 else \"12B INT8 (fits a v5e-1)\"\n",
" MODEL = TPU_REPOS.get(pick, pick)\n",
" from huggingface_hub import HfApi\n",
" hf_token = secret(\"HF_TOKEN\")\n",
" try:\n",
" info = HfApi().model_info(MODEL, files_metadata=True, token=hf_token)\n",
" weights = sum(s.size or 0 for s in info.siblings if s.rfilename.endswith(\".safetensors\"))\n",
" except Exception as e:\n",
" raise RuntimeError(f\"Can't read {MODEL} on Hugging Face ({e}). Private repo? Add HF_TOKEN to Secrets.\")\n",
" if not weights:\n",
" raise RuntimeError(f\"{MODEL} has no .safetensors files. A TPU can't run GGUF, it needs the safetensors build.\")\n",
" # vllm takes 90% of the TPU memory by default, the weights come\n",
" # out of that and the KV cache (the model's memory of the prompt\n",
" # it already read) gets the rest. ~0.3 MB a token on the 12B in\n",
" # bf16, source: the config.json math, not a benchmark\n",
" # a 16 GB chip with 12 GB of int8 weights has no room to spare.\n",
" # measured on a v5e-1: 0.48 GiB left for the cache at 0.95 in\n",
" # bf16 (1280 tokens), 0.83 GiB at 0.97 in fp8 (4864). php never\n",
" # goes under 6144. the sampler is sized for 256 parallel requests\n",
" # and she only ever has a couple (her reply, a chat title), so 4.\n",
" # 0.99 gave the cache everything and the compiled step had nowhere\n",
" # to load (3.8 MB free, vllm never budgets for that). 0.98 left the\n",
" # cache 40 MB short of 6144. 0.985 is the gap between the two\n",
" small_tpu = TPU[\"hbm\"] <= 17 * 1024**3\n",
" mem_util = 0.985 if small_tpu else 0.9\n",
" free_gib = (TPU[\"hbm\"] * mem_util - weights * 1.05) / 1024**3\n",
" if free_gib < 2:\n",
" raise RuntimeError(\n",
" f\"{MODEL} is {weights / 1024**3:.0f} GB and this TPU has {TPU['hbm'] / 1024**3:.0f} GB, \"\n",
" \"not enough room left for her to think. Pick a v6e-1 (or Kaggle's v5e-8), or the 12B INT8 build.\")\n",
" NUM_CTX = int(Context) if Context != \"auto\" else (16384 if free_gib >= 6 else 8192 if free_gib >= 3 else 6144)\n",
" # tensor parallel splits her across chips. it has to divide the\n",
" # attention heads, 1/2/4/8 do for every Gemma 4 size\n",
" tp = TPU[\"n\"] if TPU[\"n\"] in (1, 2, 4, 8) else 1\n",
"\n",
" subprocess.run([\"pkill\", \"-f\", \"vllm serve\"], check=False)\n",
" vllm_env = os.environ.copy()\n",
" if hf_token:\n",
" vllm_env[\"HF_TOKEN\"] = hf_token\n",
" # her action tags and tools need the gemma4 parsers, the reasoning\n",
" # one puts her thinking in its own field so it doesn't leak into\n",
" # the reply. text only: voice goes through whisper on your PC and\n",
" # the image/audio towers would just eat TPU memory.\n",
" vllm = subprocess.Popen([\n",
" f\"{VLLM_ENV}/bin/vllm\", \"serve\", MODEL,\n",
" \"--served-model-name\", \"jun\",\n",
" \"--host\", \"127.0.0.1\", \"--port\", \"8002\",\n",
" \"--tensor-parallel-size\", str(tp),\n",
" \"--max-model-len\", str(NUM_CTX),\n",
" \"--gpu-memory-utilization\", str(mem_util),\n",
" *([\"--kv-cache-dtype\", \"fp8\", \"--max-num-seqs\", \"4\"] if small_tpu else []),\n",
" \"--reasoning-parser\", \"gemma4\",\n",
" \"--enable-auto-tool-choice\", \"--tool-call-parser\", \"gemma4\",\n",
" \"--limit-mm-per-prompt\", '{\"image\": 0, \"audio\": 0}',\n",
" \"--download-dir\", f\"{BASE}/hf\",\n",
" ], env=vllm_env, start_new_session=True, stdout=open(f\"{BASE}/vllm.log\", \"w\"), stderr=subprocess.STDOUT)\n",
"\n",
" print(f\"Downloading {MODEL} ({weights / 1024**3:.0f} GB) and compiling it for the TPU.\")\n",
" print(\"First time takes 10-20 minutes, the compile is the slow part. Go grab a meal ...\")\n",
" started = time.time()\n",
" while True:\n",
" if vllm.poll() is not None:\n",
" print(open(f\"{BASE}/vllm.log\").read()[-3000:])\n",
" raise RuntimeError(f\"vLLM stopped. The end of {BASE}/vllm.log is above.\")\n",
" try:\n",
" urllib.request.urlopen(\"http://127.0.0.1:8002/health\", timeout=2)\n",
" break\n",
" except Exception:\n",
" time.sleep(15)\n",
" if int(time.time() - started) % 120 < 15:\n",
" print(f\" still going, {int(time.time() - started) // 60} min in ...\")\n",
"\n",
"if not INFERENCE_ONLY:\n",
" sh(\"php tools/build_lore_index.php > /dev/null 2>&1 || true\") # bakes in the Factorial Omega canon she draws on\n",
"print(f\"\\n✅ Brain installed ({NUM_CTX} tokens of context). On to Step 3.\")"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "pSuOYNRIJfDn",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 191
},
"outputId": "1d050cd1-8966-4c65-8935-eb42196fdc17"
},
"source": [
"#@title ▶️ Step 3 - Wake her up (keeps running, that's normal)\n",
"Voice = False #@param {type:\"boolean\"}\n",
"#@markdown Leave this on and she reads her replies out loud, mouth moving in time to the words (first run adds ~2 min to set the voice up).\n",
"Karaoke = False #@param {type:\"boolean\"}\n",
"#@markdown Sing with her: splits a song into a backing track and a guide vocal on Colab's GPU (needs the voice on, and a few more minutes on first run).\n",
"#@markdown Inference only ignores both: voice and karaoke run on your PC, and the phone app has its own voice.\n",
"import re, secrets, subprocess, os, time, urllib.parse, urllib.request, urllib.error\n",
"\n",
"if ACCEL == \"tpu\":\n",
" PROVIDER, UPSTREAM, PROBE = \"vllm\", \"http://127.0.0.1:8002\", \"/v1/models\"\n",
"else:\n",
" PROVIDER, UPSTREAM, PROBE = \"ollama\", \"http://127.0.0.1:11434\", \"/api/tags\"\n",
"\n",
"if INFERENCE_ONLY:\n",
" subprocess.run([\"pkill\", \"-f\", \"inference-gate.py\"], check=False)\n",
" subprocess.run([\"pkill\", \"-f\", \"cloudflared tunnel\"], check=False)\n",
" # pkill doesn't wait. a gate still on :8003 makes the new one die on bind\n",
" for _ in range(20):\n",
" if subprocess.run([\"pgrep\", \"-f\", \"inference-gate.py|cloudflared tunnel\"], capture_output=True).returncode:\n",
" break\n",
" time.sleep(0.5)\n",
" # the gate is the lock. ollama has no auth at all, so the tunnel\n",
" # never points at it directly. new key every run\n",
" gate_token = secrets.token_hex(32)\n",
" gate_env = os.environ.copy()\n",
" gate_env[\"JUN_GATE_TOKEN\"] = gate_token\n",
" server = subprocess.Popen([\"python3\", f\"{REPO_DIR}/tools/inference-gate.py\", \"127.0.0.1:8003\", UPSTREAM],\n",
" env=gate_env, start_new_session=True, stdout=open(f\"{BASE}/gate.log\", \"w\"), stderr=subprocess.STDOUT)\n",
"\n",
" # quick tunnel = no cloudflare account. pinned and checksummed\n",
" # like the ollama installer, it's a binary we run as root\n",
" CLOUDFLARED_VERSION = \"2026.10.3\"\n",
" CLOUDFLARED_SHA256 = \"053b80b0a23031a038fdbf3c073f59f82b8f1f152aa67971d7383ffafc0dbab0\"\n",
" cloudflared = f\"{BASE}/cloudflared\"\n",
" if not os.path.exists(cloudflared):\n",
" urllib.request.urlretrieve(\n",
" f\"https://github.com/cloudflare/cloudflared/releases/download/{CLOUDFLARED_VERSION}/cloudflared-linux-amd64\",\n",
" cloudflared + \".part\")\n",
" with open(cloudflared + \".part\", \"rb\") as stream:\n",
" if hashlib.sha256(stream.read()).hexdigest() != CLOUDFLARED_SHA256:\n",
" os.remove(cloudflared + \".part\")\n",
" raise RuntimeError(\"cloudflared checksum mismatch\")\n",
" os.rename(cloudflared + \".part\", cloudflared)\n",
" os.chmod(cloudflared, 0o755)\n",
" log_path = f\"{BASE}/cloudflared.log\"\n",
" subprocess.Popen([cloudflared, \"tunnel\", \"--no-autoupdate\", \"--url\", \"http://127.0.0.1:8003\"],\n",
" start_new_session=True, stdout=open(log_path, \"w\"), stderr=subprocess.STDOUT)\n",
"\n",
" tunnel = None\n",
" for _ in range(60):\n",
" time.sleep(1)\n",
" found = re.search(r\"https://[a-z0-9-]+\\.trycloudflare\\.com\", open(log_path).read())\n",
" if found:\n",
" tunnel = found.group(0)\n",
" break\n",
" if not tunnel:\n",
" raise RuntimeError(f\"The tunnel didn't come up. Peek at {log_path}\")\n",
"\n",
" # the hostname takes a few seconds to resolve after cloudflared\n",
" # prints it. checking from here means the lines below work the\n",
" # moment they show up\n",
" reachable = False\n",
" for _ in range(45):\n",
" try:\n",
" req = urllib.request.Request(tunnel + PROBE, headers={\n",
" \"Authorization\": f\"Bearer {gate_token}\", \"User-Agent\": \"JunOS-notebook\"})\n",
" urllib.request.urlopen(req, timeout=5)\n",
" reachable = True\n",
" break\n",
" except Exception:\n",
" time.sleep(2)\n",
"\n",
" if PROVIDER == \"vllm\":\n",
" lines = [\"AI_PROVIDER=vllm\", f\"VLLM_URL={tunnel}\", f\"VLLM_API_KEY={gate_token}\", \"VLLM_MODEL=jun\"]\n",
" else:\n",
" lines = [\"AI_PROVIDER=ollama\", f\"OLLAMA_URL={tunnel}\", f\"OLLAMA_API_KEY={gate_token}\",\n",
" f\"OLLAMA_MODELS_TO_PULL={MODEL}\", f\"TITLE_MODEL={TITLE_MODEL}\", f\"OMEGA_NUM_CTX={NUM_CTX}\"]\n",
" phone_link = \"junos://pair?\" + urllib.parse.urlencode([\n",
" (\"mode\", \"cloud\"), (\"name\", \"Kaggle\" if KAGGLE else \"Colab\"), (\"token\", gate_token),\n",
" (\"api\", \"openai\" if PROVIDER == \"vllm\" else \"ollama\"), (\"model\", \"jun\" if PROVIDER == \"vllm\" else MODEL),\n",
" (\"ctx\", str(NUM_CTX)), (\"url\", tunnel)])\n",
" sent_to_phone = False\n",
" if PHONE:\n",
" try:\n",
" colab_output.eval_js(f\"window.junos.postMessage({json.dumps(phone_link)}); true\", timeout_sec=10)\n",
" sent_to_phone = True\n",
" except Exception:\n",
" pass\n",
"\n",
" print(\"\\n\" + \"=\"*60)\n",
" if sent_to_phone:\n",
" print(\" 🌸 Her brain is up and the Jun OS app has the code.\")\n",
" print(\" It connects by itself and tucks this page away.\\n\")\n",
" print(\" Keep the app's \\\"Colab session running\\\" notification,\")\n",
" print(\" it's what keeps this page, and her, alive.\")\n",
" else:\n",
" print(\" 🌸 Her brain is up. Now connect Jun OS to it.\\n\")\n",
" print(\" 📱 Phone: open the Jun OS app, pick Cloud and scan this:\\n\")\n",
" qr = subprocess.run([\"php\", f\"{REPO_DIR}/tools/qr.php\", phone_link, \"svg\"], capture_output=True, text=True).stdout\n",
" if qr:\n",
" from IPython.display import SVG, display\n",
" display(SVG(qr))\n",
" print(\"\\n No camera? Copy this link, then Cloud > Paste link in the app:\")\n",
" print(\" \" + phone_link)\n",
" print(\"\\n 💻 PC: hand these lines to Jun OS, then start her.\")\n",
" print(\" Linux: ./start.sh link. Windows: Notebook link in the Jun OS\")\n",
" print(\" manager. New install: the installer's Cloud option.\\n\")\n",
" for line in lines:\n",
" print(\" \" + line)\n",
" print(\"\\n Leave this tab open while you talk to her, Colab ends\")\n",
" print(\" sessions it thinks are idle. Running this cell again makes\")\n",
" print(\" a new code and new lines, and the old ones stop working.\")\n",
" if not reachable:\n",
" print(\"\\n ⚠️ Couldn't reach the tunnel from here yet. Give it a minute,\")\n",
" print(f\" if your PC or phone can't either, peek at {log_path}\")\n",
" print(\"=\"*60)\n",
"else:\n",
" from google.colab import output\n",
"\n",
" PORT = 8000 # note: not 8080 - Colab reserves that one for itself\n",
" DOCROOT = f\"{REPO_DIR}/webapp\"\n",
" STATE_DIR = f\"{REPO_DIR}/runtime/state\"\n",
" os.makedirs(STATE_DIR, exist_ok=True)\n",
" # Her optional audio sidecar (TTS/STT, plus karaoke separation) lives on :8001.\n",
" # Docker splits those into two containers; here one process covers both.\n",
" # one shared secret between php and the audio sidecar, fresh per run.\n",
" sidecar_secret = secrets.token_hex(32)\n",
" if Karaoke and ACCEL == \"tpu\":\n",
" print(\"Karaoke needs a GPU to split songs, a TPU can't run it - skipping it this run.\")\n",
" Karaoke = False\n",
" if Voice:\n",
" print(\"Setting up her voice (downloads quietly in the background)...\")\n",
" sh(\"apt-get -qq install -y espeak-ng > /dev/null\")\n",
" sh(f\"pip -q install -r {REPO_DIR}/tts/requirements.txt\")\n",
" if Karaoke:\n",
" print(\"Setting up karaoke (this one takes a while)...\")\n",
" sh(f\"pip -q install -r {REPO_DIR}/tts/requirements-karaoke.txt\")\n",
" sidecar_env = os.environ.copy()\n",
" sidecar_env.update(STT_MAX_DURATION_S=\"120\", STT_MAX_CONCURRENT=\"1\",\n",
" SEP_MAX_DURATION_S=\"900\", SEP_MAX_CONCURRENT=\"1\",\n",
" SIDECAR_SECRET=sidecar_secret)\n",
" subprocess.Popen([\"python\", \"server.py\"], cwd=f\"{REPO_DIR}/tts\", env=sidecar_env, start_new_session=True,\n",
" stdout=open(f\"{BASE}/tts.log\", \"w\"), stderr=subprocess.STDOUT)\n",
" elif Karaoke:\n",
" print(\"Karaoke needs the voice switched on too - skipping it this run.\")\n",
"\n",
" subprocess.run([\"pkill\", \"-9\", \"-f\", \"php -S\"], check=False)\n",
" subprocess.run([\"pkill\", \"-9\", \"-f\", \"consolidation-worker.php\"], check=False)\n",
" subprocess.run([\"pkill\", \"-9\", \"-f\", \"prewarm.php\"], check=False)\n",
" url = output.eval_js(f\"google.colab.kernel.proxyPort({PORT})\")\n",
" origin = urllib.parse.urlsplit(url)\n",
" if not origin.hostname:\n",
" raise RuntimeError(\"Couldn't determine the Colab proxy hostname\")\n",
" registration_key = secrets.token_hex(16)\n",
" env = os.environ.copy()\n",
" env.update(AI_PROVIDER=PROVIDER,\n",
" OLLAMA_URL=\"http://127.0.0.1:11434\",\n",
" VLLM_URL=\"http://127.0.0.1:8002\",\n",
" VLLM_MODEL=\"jun\",\n",
" OLLAMA_MODELS_TO_PULL=MODEL,\n",
" OMEGA_NUM_CTX=str(NUM_CTX),\n",
" WATCH_FRAME_EVERY=\"2\",\n",
" TTS_URL=\"http://127.0.0.1:8001\",\n",
" SIDECAR_SECRET=sidecar_secret,\n",
" OMEGA_STATE_DIR=STATE_DIR,\n",
" OMEGA_REGISTRATION_KEY=registration_key,\n",
" # colab's proxy doesn't always pass the link's hostname\n",
" # through as Host. on a TPU runtime it's the VM's own\n",
" # t1v-n-....europe-west4-b.c.codatalab-user-runtimes.internal\n",
" # and we can't ask for it, this kernel is a docker\n",
" # container whose hostname is its container id. so the\n",
" # leading-dot rule takes all of .internal, a reserved\n",
" # TLD nobody can register, so no public site can rebind\n",
" # to it. php only listens inside the container anyway\n",
" OMEGA_ALLOWED_HOSTS=\",\".join([origin.hostname, \".internal\", \"localhost\", \"127.0.0.1\"]),\n",
" OMEGA_ALLOWED_ORIGINS=f\"{origin.scheme}://{origin.netloc}\")\n",
" if PROVIDER == \"ollama\":\n",
" env[\"TITLE_MODEL\"] = TITLE_MODEL\n",
" subprocess.run([\"php\", \"-r\", f\"require '{DOCROOT}/api/lib/bootstrap.php'; db();\"], env=env, check=True)\n",
" subprocess.Popen([\"php\", os.path.join(DOCROOT, \"api\", \"consolidation-worker.php\")], env=env, start_new_session=True,\n",
" stdout=open(f\"{BASE}/memory.log\", \"w\"), stderr=subprocess.STDOUT)\n",
" subprocess.Popen([\"php\", os.path.join(DOCROOT, \"api\", \"prewarm.php\")], env=env, start_new_session=True,\n",
" stdout=open(f\"{BASE}/prewarm.log\", \"w\"), stderr=subprocess.STDOUT)\n",
" server = subprocess.Popen([\"php\", \"-S\", f\"127.0.0.1:{PORT}\", \"-t\", DOCROOT,\n",
" f\"{REPO_DIR}/tools/php-router.php\"], env=env, start_new_session=True,\n",
" stdout=open(f\"{BASE}/php.log\", \"w\"), stderr=subprocess.STDOUT)\n",
" time.sleep(2)\n",
"\n",
" print(\"\\n\" + \"=\"*60)\n",
" if url:\n",
" print(\" 🌸 Jun's waiting for you. Open this link:\\n\")\n",
" print(\" \" + url + \"\\n\")\n",
" print(\" It asks for a registration key the first time, it's this one:\")\n",
" print(\" \" + registration_key + \"\\n\")\n",
" print(\" Make an account (you'll confirm you're an adult) and say hi.\")\n",
" print(\" She loads in the background, so the very first reply can\")\n",
" print(\" take 1-2 minutes. Leave this tab open while you talk to her.\")\n",
" else:\n",
" print(\" Couldn't get a link this time. Re-run this cell.\")\n",
" print(\"=\"*60)\n",
"\n",
"# colab and kaggle end a session once no cell is running, so this one\n",
"# never finishes. it follows her logs instead, minus the polling GETs\n",
"from IPython.display import Pretty, display\n",
"NOISE = re.compile(r\" (Accepted|Closing)$|\\]: (GET|HEAD) |\\| (GET|HEAD) |\\\"(GET|HEAD) /|^(GET|HEAD) 2\\d\\d$\")\n",
"logs = [f\"{BASE}/{name}.log\" for name in\n",
" ([PROVIDER, \"gate\"] if INFERENCE_ONLY else [PROVIDER, \"php\", \"memory\"] + ([\"tts\"] if Voice else []))]\n",
"offsets = {log: os.path.getsize(log) if os.path.exists(log) else 0 for log in logs}\n",
"started = time.time()\n",
"status = display(Pretty(\"\"), display_id=True)\n",
"try:\n",
" while server.poll() is None:\n",
" for log in logs:\n",
" if not os.path.exists(log):\n",
" continue\n",
" if os.path.getsize(log) < offsets[log]:\n",
" offsets[log] = 0\n",
" with open(log, \"rb\") as stream:\n",
" stream.seek(offsets[log])\n",
" chunk = stream.read()\n",
" offsets[log] += chunk.rfind(b\"\\n\") + 1\n",
" for line in chunk[:chunk.rfind(b\"\\n\") + 1].decode(\"utf-8\", \"replace\").splitlines():\n",
" if line.strip() and not NOISE.search(line):\n",
" print(f\"{os.path.basename(log)[:-4]:>7} | {line[:300]}\")\n",
" up = int(time.time() - started) // 60\n",
" status.update(Pretty(f\"Up {up // 60} h {up % 60:02d} min. This cell runs forever on purpose, it's what keeps the session alive. Don't stop it.\"))\n",
" time.sleep(5)\n",
" print(f\"\\n⚠️ {'The gate' if INFERENCE_ONLY else 'php'} stopped. Re-run Step 3 to bring her back.\")\n",
" if INFERENCE_ONLY and os.path.exists(f\"{BASE}/gate.log\"):\n",
" print(\"\".join(open(f\"{BASE}/gate.log\", errors=\"replace\").readlines()[-15:]))\n",
"except KeyboardInterrupt:\n",
" print(\"Stopped following the logs. She's still running, but Colab now counts the session as idle and ends it sooner.\")\n"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "evvhNleNJfDn"
},
"source": [
"## When things go sideways\n",
"\n",
"- **Started from the phone app and it never connected:** Step 1 should say \"Started from the Jun OS app\". If it doesn't, set `Mode` to Inference only, run Step 1 and Step 3 again, then copy the link Step 3 prints and tap **Paste link** in the app.\n",
"- **The phone app says Google won't let you sign in:** tap **Google won't sign me in here**. The notebook opens in your browser, set `Mode` to Inference only and paste Step 3's link into the app. The browser has to stay open that way, the app can't keep it alive for you.\n",
"- **The link won't open / says \"unknown host\":** give it ~30 seconds and reload, or just re-run Step 3 for a fresh one.\n",
"- **The page just says `invalid host`:** Colab's proxy sent a name she doesn't answer to. `!grep \"invalid host\" /content/php.log` shows which one, report it so it can be added.\n",
"- **Her first reply is slow:** totally normal, she's loading into memory. Everything after that is quick.\n",
"- **Your account and chats vanished:** Colab wipes the whole session when it ends, so nothing carries over between runs. For a Jun that actually remembers you, run her [on your own machine](https://github.com/efficiencyx/JunOS#get-her-running), or keep her there and use this notebook in inference-only mode.\n",
"- **Inference only, and your PC or phone says she's unreachable:** the session ended or Step 3 ran again, both change the URL and the key. Re-run Step 3 and paste the new lines (phone app: scan the new code, or tap Start a new Colab session).\n",
"- **Step 2 says the model doesn't fit the TPU:** the full-precision 12B needs a v6e-1 or bigger. On a v5e-1 leave `TPU_Model` on `auto`, or pick the INT8 build.\n",
"- **vLLM stopped on a TPU:** the end of `vllm.log` is printed above the error. A model vLLM doesn't know yet, or a context too big for the memory left (`Context` back to `auto`), are the usual two.\n",
"- **Step 3 never finishes:** on purpose. Colab and Kaggle end a session once no cell is running, so it stays busy printing her logs. Stopping it leaves her running, the session just starts counting as idle.\n",
"- **Want to see what she's thinking?** Step 3 prints the interesting lines as they come. The full logs live in `/content/` (`/kaggle/working/` on Kaggle): `ollama.log` or `vllm.log`, `php.log`, `tts.log`, `gate.log`, `cloudflared.log`. Stop Step 3, then peek at one with e.g. `!tail -n 40 /content/php.log`, and run Step 3 again after (new link, and in inference-only mode new lines to paste)."
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"gpuType": "T4",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}