Hmm… personally, my take is basically: “There are small models that will run on an M2 with 8 GB, but if the actual goal is coding, I would recommend an API/cloud service on those specs.” More specifically:
I would separate “Can I run a model locally?” from “Will this make a useful coding agent?” Those are fairly different questions.
For an 8 GB M2 Mac, I would not make Qwen3.8-27B my practical local target. Qwen describes it as a dense 27B model, and the normal MLX Community 4-bit conversion is already about 16 GB on disk, before accounting for macOS, the inference runtime, caches, agent history, etc.
That does not mean nobody can ever make it start with some unusual low-bit/offload/swap setup. It means I would not build an everyday coding workflow around that on an 8 GB machine.
On the other hand, small local models are absolutely worth trying. For example, the MLX 4-bit Qwen3.5-4B artifact is only about 3 GB, and Qwen3.5-4B explicitly supports coding/tool-oriented use.
There is even a recent OpenJarvis issue from an M2 / 8 GB Mac where ollama run qwen3.5:4b itself worked. The surrounding OpenJarvis invocation failed for a different integration/routing reason. That is only one report, not a performance guarantee, but it is a useful reality check that 4B-class local inference on this hardware is not hypothetical.
My default decision tree would therefore be roughly:
I mainly want to USE an agentic coding assistant
→ start with a hosted coding agent such as Codex
I want inexpensive hosted access to open-weight models
→ Groq GPT-OSS / DeepSeek / OpenRouter, depending on what I need
I specifically want privacy, offline use, or to learn local inference
→ try a 2–4B quantized model locally
I want both
→ small local model for cheap/light tasks
+ hosted stronger model for difficult repository work
If I were starting from your machine today, I would probably try the hosted route first for actual programming, and separately keep a small local model as an experiment.
That lets you learn what an agentic coding workflow is supposed to feel like without first making “fit the model into 8 GB” the main engineering project.
If open weights are not a requirement, Codex is one straightforward starting point. It is currently included across ChatGPT plans, including Free and Go with plan-dependent limits, and has desktop, CLI, IDE, and web entry points.
If you prefer open-weight models via API, there are also surprisingly inexpensive options. For example, Groq’s GPT-OSS 20B currently has very low token pricing, a large context window, and tool/reasoning support. Groq also currently provides a free-tier allowance; I would check the live rate-limit page rather than treating today’s limits as permanent.
DeepSeek V4 Flash / Pro are another inexpensive hosted option. Both currently support tool calls and a large context window. One useful distinction at the moment is that the Responses API is listed for V4 Flash but not V4 Pro, so I would check the current integration matrix rather than assume the two are drop-in equivalents.
OpenRouter’s free models can also be useful for learning and comparing models, although I would treat free routing more as an experiment facility than as a reproducible production backend.
Why 4B working locally does not automatically mean a full coding agent will work well
There are really several layers involved:
model
↓
inference runtime
↓
API / chat template / tool protocol
↓
agent harness
↓
repository search / file editing / shell / tests
A failure at the bottom can easily look like “the model is bad.”
The M2/8GB OpenJarvis report above is a nice small example: direct ollama run qwen3.5:4b worked, while the surrounding agent path did not.
There are other examples of the same general boundary. For example, Qwen Code issue #9438 documented a case where the request constructed after a tool call had lost important conversation state. From the outside, that kind of failure can look like the model suddenly became confused, when the actual problem is in the harness/request path.
There is also broader research evidence that the agent scaffold itself matters substantially.
SWE-agent framed this as the Agent-Computer Interface (ACI) problem: how repository navigation, editing, testing, and feedback are exposed to the model affects what the model can accomplish.
More recently, SWE-Bench Mobile reported very large differences between agent systems using the same underlying model.
That benchmark is not a test of an M2 Mac or Qwen3.5-4B, so I would not transfer its absolute numbers here. But it is strong evidence against treating:
model benchmark score
as identical to:
coding-agent quality
A more useful mental model is:
model
+ context selection
+ agent loop
+ tools
+ edit protocol
+ test feedback
So if a small local model behaves strangely inside an agent, I would avoid immediately concluding that it simply needs more parameters.
What I would realistically expect from a 2–4B local model
I would think about local coding capability as a ladder:
1. explain code / answer a question
2. generate a small function
3. make a bounded one-file edit
4. search/read → edit → run one targeted test
5. multi-file repository work
6. long autonomous repository-agent loop
On 8 GB, I think 1–3 are very reasonable things to experiment with using a current 2–4B quantized model.
Level 4 can also be interesting if the harness keeps the task small and gives the model useful test feedback.
I would be much more cautious about assuming 5–6 will be reliable simply because the model loads.
Qwen3.5-4B illustrates part of the distinction. It has real coding/tool capabilities, but Qwen also discusses very large context settings for preserving its more complex reasoning behavior.
That is a very different deployment target from:
“4B quantized model running comfortably inside an 8 GB Mac alongside an IDE, repository, tool history, and KV cache.”
So the small model can still be very useful at shorter context; I just would not assume that a low-memory deployment reproduces every long-context agent setup demonstrated for the original model.
A second compact option worth looking at is Granite 4.1 3B. IBM documents coding tasks, function calling, agent/tool use, and fill-in-the-middle capabilities for it. Again, that does not make it a substitute for a large coding model on every repository, but it shows how broad the useful 3–4B class has become.
For 8 GB, a lightweight coding workflow matters a lot
Aider is an interesting example because it does not solve repository awareness by blindly dumping the entire repository into the prompt.
Its repository map extracts useful information such as files, classes, functions, and signatures, then ranks what to include within a limited token budget.
Aider also notes in its FAQ that weaker models can sometimes become more confused by repository-map information.
So:
more context != automatically better context
is worth keeping in mind.
Its Ollama documentation is also useful for local setups. It specifically warns about context truncation: if old context is silently discarded, a model can appear to “forget” files or instructions when the real issue is that those tokens no longer reached the model.
For a small local model, I would prefer something conceptually like:
locate the relevant code
↓
read only the relevant file/function
↓
make one bounded change
↓
run the narrowest useful test
↓
return the actual failure if it fails
This is also why Agentless is an interesting reference.
Instead of giving a model an unrestricted long agent loop, it decomposes software repair into roughly:
localization
↓
repair
↓
patch validation
Agentless used much stronger models than the 4B setup discussed here, so its benchmark numbers should not be transferred to an 8 GB Mac. The useful part here is the decomposition strategy.
Likewise, mini-SWE-agent is interesting because it deliberately keeps the harness very small. Its local-model documentation even shows local-model configurations down into the small-model range.
That does not prove those tiny models are excellent software engineers. It shows that a huge agent framework is not a prerequisite for experimenting with agentic coding.
Aider’s architect/editor approach is another example of useful role separation:
architect
↓
reason about what should change
editor
↓
turn that decision into precise file edits
You do not need to build that architecture yourself as a beginner.
I mainly mention it because one enormous model does not necessarily need to perform every part of an agentic workflow.
Where small models are genuinely getting interesting
There is encouraging work here, but I would interpret it as:
“Small models can become very useful specialists.”
rather than:
“4B now replaces every large coding model.”
For example, SWE-Spot explicitly starts from the problem that small models have difficulty generalizing across unfamiliar, complex repositories.
It then explores repository-specialized 4B experts and reports that a small specialist can become very competitive inside the environment it knows well.
That suggests a future local coding system may look less like:
one tiny model does everything
and more like:
small general model
+ repository/task specialist
+ search/localization
+ compact repository context
+ tests/verifiers
+ stronger fallback when required
Another particularly relevant project is CodeScout, which specializes small models for code localization—finding the file/class/function that is probably relevant to an issue.
Its 4B model card is also quite explicit about the boundary: it is intended for localization, not as a complete software-repair model.
That is a useful pattern for low-resource systems:
small model:
find where the problem probably is
stronger model or another stage:
reason about the difficult change
tests:
verify what actually happened
Tool calling is similar. Small models can be trained to call tools well, but tool-call skill is not identical to repository-level coding ability.
So I would judge small models role-by-role rather than only by parameter count.
A practical hosted/API comparison
I would separate the coding-agent product/harness from the model provider.
1. “I just want to start coding with an agent”
A finished coding-agent environment such as Codex is probably the lowest-friction route if open weights are not a requirement.
You avoid choosing:
- quantization
- context/cache settings
- chat templates
- tool parsers
- inference servers
- repository tooling
before you have even tried the workflow.
You can first learn what it feels like to let an agent inspect a repository, edit files, and run checks.
Then, if local inference is interesting to you, you have a useful baseline to compare against.
2. “I want open-weight models and cheap API inference”
Groq + GPT-OSS is an unusually inexpensive example right now.
GPT-OSS 20B currently provides a large context window and support for tool-oriented workflows, while avoiding the 8 GB local-memory limit.
Groq also has a free-tier/rate-limit page, so it can be quite useful for initial experiments.
3. “I want DeepSeek”
DeepSeek V4 Flash and Pro currently both support tool calls and a 1M advertised context window.
At the moment there is an important API distinction:
- V4 Flash: Responses API supported
- V4 Pro: Responses API not currently listed as supported
So I would check the current table before choosing an integration.
DeepSeek also has DeepSeek Harness, which makes the model/harness distinction unusually explicit:
Agent = Model + Harness
Models, tools, skills, sessions, sandboxes, storage, loops, and UI are separate pieces.
The project currently describes itself as a developer preview with breaking changes expected, so I would treat it more as an interesting playground for understanding/building agents than as the simplest beginner route.
4. “I want free APIs”
OpenRouter’s free models are useful for learning and quick comparisons.
For debugging an agent, though, I would prefer a specific fixed model over a router that may choose a different free model between requests.
Otherwise:
the second attempt worked
could mean either:
my prompt/harness improved
or simply:
I got a different backend model
A cheap fixed endpoint gives a cleaner comparison.
A cheap way to find out whether the local model or the agent is the bottleneck
If you do want to experiment locally, I would avoid starting with a giant benchmark suite.
One short progression gives a lot of information:
A. direct local inference
↓
B. one small coding question
↓
C. one read/search tool call
↓
D. one bounded file edit
↓
E. one targeted unit test
If something fails, one useful sanity check is to run the same task through the same harness once with a known stronger hosted model.
The branches are fairly informative:
direct local inference works
but agent + local model fails
→ inspect harness / context / tool protocol /
integration before blaming raw model inference
hosted model succeeds
but local model repeatedly fails
→ local model capability / quantization /
context / local runtime become stronger suspects
both local and hosted models fail
→ task specification / harness / tools /
environment become stronger suspects
That is not a perfect controlled experiment because different providers can use different templates and tool implementations.
But it is a cheap way to avoid spending hours solving the wrong problem.
I would also repeat any tiny local coding test a couple of times before deciding that the model either “can” or “cannot” perform it. Small models can show substantial run-to-run variation once reasoning and tool selection are involved.
Will weak computers become much better at this?
I would say probably yes, but I would not frame it only as:
today's 27B capability
↓
tomorrow's 4B model
Several independent things are improving:
- small-model coding/reasoning quality
- quantization
- Apple-Silicon inference runtimes
- retrieval and repository mapping
- context compression
- specialized small models
- tool interfaces
- test/verifier loops
- routing difficult steps to stronger models
Context handling in particular matters for agents because every tool call can add observations and history.
For example, ACON studies compression of agent observations/history and reports substantial reductions in peak context use while retaining much of the task performance; it also reports improvements for smaller models in some long-horizon settings.
That is research evidence, not a guarantee for a particular coding tool, but it illustrates why future capability on an 8 GB computer is partly a software/system-design problem, not just a parameter-count problem.
Likewise:
- SWE-Spot explores repository specialization.
- CodeScout separates localization into a specialist.
- Aider’s repo map reduces how much repository information the main model needs at once.
- Agent benchmarks increasingly show that the scaffold around the model matters substantially.
So I do expect low-memory machines to become increasingly useful for agentic work.
I just would not wait for:
“a frontier-size general coding agent completely inside 8 GB”
before trying the workflow today.
A hybrid design is already reasonable:
local small model
→ explanation
→ search assistance
→ summarization
→ simple transforms
→ maybe bounded edits
hosted stronger model
→ difficult reasoning
→ unfamiliar repositories
→ long multi-step work
tests / git / deterministic tools
→ verification
That architecture can improve over time simply by replacing one component.
One small safety/operational note
Once an AI assistant can run shell commands and modify files, “local” does not automatically mean “safe.”
For a first experiment I would keep the project in Git, inspect the diff, and keep write/shell/network operations approval-gated until I understood what the harness actually does.
The useful trust boundary is roughly:
model proposes an action
↓
harness checks whether it is allowed
↓
tool performs the action
↓
git/tests let me inspect the result
That is better than relying on the model itself to decide which machine operations are safe.
And if you send private/proprietary source code to a hosted provider, that becomes a separate provider/data-policy question from model quality or price.
So, if this were my M2 8 GB machine, I would probably do both:
- Use a hosted coding agent/API for real coding work, so the 8 GB RAM limit is mostly irrelevant.
- Keep a 2–4B quantized model locally and experiment with small, well-bounded coding tasks.
That gives you something useful now while still letting you learn the local/open-model side.
I would not spend much effort trying to force Qwen3.8-27B into 8 GB first. If the local experimentation turns out to be the part you enjoy, there is already a lot to explore with small models, lightweight harnesses, repository maps, role separation, and test-driven agent loops before hardware becomes the only interesting variable.
