Sprint 26 Planning

[gabrielle-ong]

1.0.4 scope: https://github.com/janhq/cortex.cpp/milestone/25

Sprint 25 In progress / QA + Release next week

• roadmap: Cortex API supports /threads, /messages #1567
• docs: Doc fixes (tools, engines, API Reference is missing sidebar items) #1675
• roadmap: Implement Remote engine #1662
• feat: Automate nightly e2e testing for cortex.cpp against cortex.llamacpp pre-release #1729
• release name alignment to upstream llamacpp janhq/cortex.cpp#303

---

Sprint 25 Not yet started + Sprint 26 stories

Jan: Engine Management, Hardware Settings

Jan Bugs that should be solved (requires QA)

• bug: Jan::Error: Load model failed with error TypeError: fetch failed #1467
• bug: insufficient handling of insufficient memory #1457

Jan Bugs that needs investigation (to prioritise)

• bug: Mismatch between Jan UI & model configuration options jan#4035
  Sub tasks:

• Wrong NGL maximum settings (e.g., NGL number limits at 30 layers but the user can set it to 100 in Jan UI): https://discord.com/channels/1107178041848909847/1306758623325851689/1307651904171540512

• bug: Max context length automatically defaults to 4096 tokens #1668

• bug: OpenRouter API max token limit too low in UI (1024) compared to model capability (128k) jan#4014

• bug: Inference with NVIDIA GPU stop working after resuming from sleep on Linux #1459 - computer going to sleep

• bug: Arch Detector Causing Special Characters in GPU Name in settings.json #1140

• bug: Cortex Engine crashes - Doesn't start any local models after the latest update #1469

• bug: Tokens per second calculation is wrong. #1099, epic: cortex.cpp Hardware Benchmarking + Backend Infra #985 Tokens per second calculation:

Need technical help understanding:

• feat: set up HTTPS for Custom Models #1097: is this fixed by proxy?
• roadmap: mmap for keeping Model in VRAM when Flash Attention is used #1717

Jan x Cortex implements Hardware Detection

• bug: Cortex-cpp continues to have 1 layer offload to CPU while using GPU #1104
• bug: Models loading onto wrong GPU in multi-GPU setup #1468

Cortex supports new Engines

• docs: how to build an Engine #1731
• roadmap: Engines are dylibs and are self-contained #1732

New Features

P0: Model sources - needed for Jan Hub

• planning: Cortex API supports /model/sources #15

P0: Jan x Cortex supports Python

• janhq/jan-internal#8

Assistants

• planning: Cortex API supports basic /assistants (Jan status quo equivalent) #1573
• Planning: RAG = Assistant specific File Search janhq/cortex.cpp#20

QA

• planning: Cortex Automated QA Strategy #1565
• epic: unitest for cortex cpp #1069
• https://github.com/janhq/qa/issues/3

Cortex UX Improvements

• planning: Cortex UX improvements [WIP] #1685
• epic: cortex models update <MODEL_ID> #1060
• bug: Interrupted cortex update does not stop gracefully #1656

---

Sprint 27 / beyond

Support Ichigo & TTS & STT -> Sprint 27, blocked on Python

• Jan Parent issue: roadmap: Jan supports Local Voice Mode w/ Ichijo jan#3488
  • epic: cortex.cpp support Ichigo & TTS #1247
  • fix wrong include path #23
  • epic: Cortex supports Audio (STT) #743

RAG

• planning: Cortex API supports Retrieval Tool #1597

Supports Vision, upstream llama.cpp -> Move to Sprint 27

• roadmap: llamacpp-engine to align with llama.cpp upstream #1728
• planning: Supporting vision model (Llava and Llama3.2) #1493

Unprioritised / Icebox

• epic: Cortex API supports /model/presets #1185
• feat: Cortex has reRank models #1101
• epic: cortex.cpp Hardware Benchmarking + Backend Infra #985
• CI for VNNI support janhq/cortex.cpp#60

Jan & Cortex supports Multi GPU -> Create parent Jan issue

• planning: prioritize GPUs with CUDA_VISIBLE_DEVICES #1679
• Add Multi-GPU Support for LlamaCpp Engine #1391
• Fixes Jan multiple GPUs #1458

Docs / Website:

• docs: idea: Using Cortex & Jan #1736
• Decision: Are we publicly marketing Cortex or focusing on Jan?

[gabrielle-ong]

Suggested Jan Epics (as parent issues for Cortex)

cc @imtuyethan @dan-homebrew @0xSage

3. Jan & Cortex fixes Vulkan acceleration

• Need Eng planning / investigation
• bug: Failed to Load Model with AMD GPU and Vulkan #1525
• bug: AMD RX 7900 XTX Incompatibility Even with Vulkan Mode Enabled jan#3874

4. Jan & Cortex supports Multi GPU

• Add Multi-GPU Support for LlamaCpp Engine #1391
• planning: prioritize GPUs with CUDA_VISIBLE_DEVICES #1679
• Will fix: bug: some models failed to load if many GPU are selected #1458

[gabrielle-ong]

Closing after sprint planning discussion, now on kanban boards