Local AI on a 6 GB GTX 1060: what worked and why I ended up removing it

I have a Linux laptop (CachyOS) with 16 GB of RAM and an NVIDIA GTX 1060 with 6 GB, a graphics card from 2016. And on a Raspberry Pi I had an AI agent connected to a cloud service on a free plan with 50 requests a day, which an agent burns through in no time.

The idea seemed good: run a local model on the laptop and use it as a fallback when the free requests ran out. I set it up, measured it… and removed it the same day. Here’s everything, because the numbers and the stumbles might save you an afternoon.

The CUDA trap with older cards

First attempt: install Ollama from the distro package (ollama-cuda). It installed fine, loaded the model… and was painfully slow. Looking at the logs, Ollama had silently skipped the GPU and was running on the CPU.

The reason: that package was built with CUDA 13, and CUDA 13 no longer supports the Pascal architecture (the GTX 1060’s, compute capability 6.1). There’s no clear warning: it just doesn’t use it.

The fix was the official Ollama binary, which ships with CUDA 12 libraries. You can even extract it into your home folder (~/.local) without admin rights. With that, the GPU finally started working.

If you have a 10-series NVIDIA card (Pascal) and Ollama is weirdly slow, check which CUDA version your package ships before fighting with anything else.

Another detail: even if the model fits entirely in VRAM, Ollama is conservative and sometimes leaves layers on the CPU. The num_gpu 99 parameter tells it to put everything it can on the GPU.

The numbers

I tried two sizes of Qwen3, an open model that handles tools reasonably well:

Model	Context	Speed	Runs on
Qwen3 8B	default	25.4 tokens/s	100% GPU
Qwen3 4B	64,000 tokens	31.3 tokens/s	100% GPU
Qwen3 8B	64,000 tokens	15.9 tokens/s	partly CPU

For chatting, 25 tokens per second is plenty: you read slower than it writes. As a local chat, it worked well.

Where it broke: the agent

The problem wasn’t the model but what I wanted it for. The agent I was using requires at least 64,000 tokens of context from any model. And its system prompt alone, with the descriptions of all its tools, takes about 11,500 tokens before you’ve said anything.

With 6 GB of VRAM, the only thing that fit entirely on the GPU with 64k of context was the small 4B model. It was fast, but it made up the syntax of tool calls. The agent asked it to use a tool and the model answered in a malformed format, missing the tool name. Three times in a row.

For an agent, calling tools correctly isn’t a bonus: it’s the whole job. A model that fails at it is no use, however fast it is.

The 8B at 64k would probably have been good enough, but at 15.9 tokens/s with part of it on the CPU, the experience was no longer worth it.

What NOT to do

I also tried enabling unified memory (GGML_CUDA_ENABLE_UNIFIED_MEMORY=1), which lets you use regular RAM as if it were GPU memory. In theory, that lets a bigger model fit. In practice, it crashed the Ollama server. Not an option on this machine.

And a lesson in sharing: I ran heavy tests while a session was actively using the laptop. The machine stopped responding for a while. If you’re going to load up a computer someone is using, warn them first (even if that someone is you in another window).

Why I removed it

By the end of the afternoon I had:

• A local chat that worked fine, but that I didn’t need.
• A fallback for the agent that didn’t work.
• A hotter laptop with less battery life.

So I deleted it all. “Not worth keeping if it doesn’t work well” was the verdict. Setting something up and removing it the same day isn’t a failure if you keep the data.

If you want to try

• For chatting or summarising text locally, a GTX 1060 with a 7-8B model is more than enough.
• For agents with many tools and long context, 6 GB falls short. Look for at least 12 GB of VRAM, or use a cloud service.
• Use the official Ollama binary if your card is Pascal.
• Measure before deciding: ollama run model --verbose tells you tokens per second.