Running Qwen3.8-27B on a Mac mini and an RTX 3090

I ran the Q4_K_M quantized version of Qwen3.8-27B on an M4 Mac mini and a custom PC equipped with an RTX 3090. To state the conclusion first, the 32GB Mac mini was able to secure a 128K context and launch without issues. However, the generation speed was approximately 5.6 tokens/sec, showing a significant difference compared to the approximately 40 tokens/sec of the RTX 3090 environment.

This result clearly demonstrates that "whether the model fits in memory" and "whether the loaded model runs fast" are separate problems. Capacity determines feasibility, while bandwidth primarily determines generation speed.

The Model Run This Time

The target is the GGUF version of the official Qwen3.8-27B quantized to Q4_K_M. Below, we treat Qwen3.8-27B Q4_K_M files in the 16GB range as the same hardware class, including differences of tens to hundreds of MB depending on the distributor.

Q4_K_M is a quantization method in the K-quants family that allocates precision based on importance, rather than mechanically rounding all weights to 4-bit. The BF16 weights of the 27B model require approximately 55GB, but with Q4_K_M, they shrink to the 16GB range. It is thanks to this compression that it could realistically be loaded into 32GB of unified memory.

Operating Environment

Mac mini

On Apple Silicon, the CPU and GPU share the same unified memory. Therefore, there is no fragmentation where "it fits in system RAM but not in GPU VRAM," allowing the entire 16GB-range model to be offloaded to Metal. On the other hand, that 32GB is shared with browsers and regular desktop apps. Since it is not dedicated VRAM, the remainder cannot all be used exclusively for inference.

Custom PC with RTX 3090

The 24GB VRAM of the RTX 3090 can accommodate the entire 16GB-range Q4_K_M model. Since the system memory is also 64GB, a major difference from the Mac mini is that GPU-dedicated memory and memory used by everyday apps can be separated.

Launching "Only When Needed" with Nix

On the Mac mini, llama.cpp's llama-server was managed via Nix / Home Manager. The canonical configuration is placed in nix-config, and the following conditions are given as Mac mini-specific settings.

• Offload all layers to Metal
• Context limit is 131,072 tokens (128K)
• Quantize K/V of the KV cache to Q8_0
• Enable Flash Attention
• Parallel slots: 1
• Expose OpenAI-compatible API at 127.0.0.1:8080
• Do not auto-start on login; launch only when needed with qwen38 start

Auto-start was avoided because the Mac mini is not dedicated to LLMs. Naturally, when the model is loaded, the available memory for everyday use decreases. If using browsers or development environments concurrently, launching on-demand is easier to manage than keeping it resident.

Since it is OpenAI-compatible, the client side does not require special implementations. By changing the connection destination of the OpenAI SDK to local, it can be called in the same way from custom tools, chat UIs, editor extensions, agents, etc.

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "local-model",
    "messages": [{"role": "user", "content": "こんにちは"}],
    "reasoning_effort": "none"
  }'

The chat template used this time enables thinking mode by default, so reasoning_effort: "none" was specified when short, normal responses were desired. Without specifying it, the output budget might be exhausted by internal thinking tokens.

Measured Results

The perceived difference when running the same Q4_K_M model was clear.

On the Mac mini, loading the model with the 128K setting took approximately 17 seconds. Memory pressure during short generation remained green, and normal screen operations were possible. Approximately 15GB of the model was mapped to the process, and the total appeared to be on the scale of about 20GB. However, this is the state with short prompts. If the conversation history is actually filled up to near 128K, the KV cache will grow, increasing memory consumption and Prefill wait times.

Long-form Generation of 2,048 Tokens

Not just short responses, but the same Japanese prompt was given to both machines, and generation was run up to the limit of 2,048 tokens. Conditions were aligned with temperature: 0, reasoning_effort: none, and parallelism 1.

The actual file used for performance measurement is a derivative Q4_K_M (16.81GB) of Qwen3.8-27B. The difference from the standard Q4_K_M (local file is 16.46GB) is approximately 2.1%, and the architecture and quantization class are the same. Therefore, in this article, it is not used for comparing response content, but treated as a reference value for hardware performance dominated by model size and memory bandwidth.

The speed ratio was approximately 7.59x. The Mac mini went from approximately 5.6 for short texts to 5.44 tokens/sec, and the RTX 3090 from approximately 40 to 41.30 tokens/sec. At least for continuous generation up to 2,048 tokens, no clear speed degradation due to longer output was observed. On the RTX 3090 side, approximately 40.3 tokens/sec was maintained even at the point of generating 7,773 tokens in a separate run.

What was measured here is long output, not long input where materials close to 128K are loaded first. This time, the context ultimately used remained at approximately 2,139 tokens combined for input and output. Since Prefill time and KV cache capacity affect long inputs separately, one cannot conclude that "the experience is the same across the entire 128K range."

Also, this is not a strict comprehensive benchmark, but an observation based on local operational settings. It does not include comparisons of model quality, long-form Prefill near 128K, power consumption, or concurrent execution counts.

Why Is There a Difference of About 7x?

Initially, the difference in GPU computational performance seems to be the main cause. However, LLM Decode reads weights every time a token is generated, so it is strongly influenced by memory bandwidth rather than computational power. The rough upper limit can be estimated by the following formula.

tokens/sec ≈ memory bandwidth ÷ model size

Simple calculation with the 16.8GB model actually measured this time yields the following.

The bandwidth ratio is approximately 7.8x, and the measured speed ratio is approximately 7.1x. Of course, in reality, factors such as cache, kernel efficiency, and per-tensor quantization formats come into play, so it does not follow the formula exactly. Nevertheless, it fits surprisingly well as a first-order approximation to explain the difference this time.

The M4 has a wide 120GB/s bandwidth for an integrated GPU, but the RTX 3090's GDDR6X has approximately 936GB/s. The advantage is not that "unified memory is fast," but that "the model can be loaded into a large shared area without copying between CPU and GPU." The sequential generation speed after loading cannot escape the bandwidth limit of that shared memory.

It Didn't Get Faster Even with Smaller Context

Comparing short responses with 8K and 128K, the speeds were almost the same at approximately 5.58 and 5.65 tokens/sec. What is easily confused here is "context limit" and "actual context length in use."

Even if the limit is set to 128K, short conversations do not perform 128K worth of calculations per token. The 16GB-range model weights read every time in Decode also remain unchanged. Therefore, the speed of short generation hardly improves.

On the other hand, if the actual input becomes longer, the situation changes. Prefill, which processes the initial input, becomes heavier, and the KV cache to be saved increases proportionally to the length. In other words, the effect of making the context smaller is mainly to "suppress maximum memory usage" and "not allow extremely long inputs," not to increase the tokens/sec of short responses.

How Does It Compare to GPT-5.6 Luna?

To consider the value of building a new local LLM, it is necessary to compare it with the case of using a cheap cloud model for similar purposes. Here, OpenAI's GPT-5.6 Luna was used as the comparison target.

Detailed official benchmark values for Luna alone are not publicly available. According to OpenAI, Luna is a cost-focused, high-volume processing model that roughly corresponds to the nano tier in the past GPT-5 family. Therefore, the following is a comparison using GPT-5.4 nano, which has published values, as a reference for positioning, rather than Luna itself.

Numerically, Qwen3.8-27B is higher, but the evaluation conditions are not unified. Also, the Q4_K_M run this time is a 4-bit quantized version, and there is no guarantee it will produce the same scores as the official BF16 version. On the other hand, Luna is a newer generation, and it cannot be definitively stated that it has the same performance as GPT-5.4 nano based solely on the information that it is nano-equivalent. Therefore, I think it is safe to view "Qwen Q4_K_M and Luna are close classes that may vary depending on use case."

The difference is greater in the surrounding environment than in the model itself. Luna has a maximum context of 1.05M tokens and can utilize Web search, file search, Code Interpreter, Computer Use, MCP, etc. Tools can also be custom-built and connected to local Qwen, but their construction and maintenance must be done by oneself.

Measuring Luna from This Mac mini

To go beyond an estimated comparison, I called Luna through the Responses API from this Mac mini over the internet. I used the same type of Japanese prompt as the local long-form generation test, with reasoning.effort: none and a maximum output of 2,048 tokens.

In the non-streaming measurement, Luna processed 119 input tokens and 2,048 output tokens in 19.21 seconds, giving an effective end-to-end speed of 106.59 tokens/sec as observed from this machine. In a separate streaming run, TTFT (Time to First Token) was 2.33 seconds, streaming after the first text arrived took 18.58 seconds, and total elapsed time was 20.91 seconds.

The Luna result includes network round trips from this machine to OpenAI as well as API-side waiting time. Even so, it was approximately 2.6 times faster than the RTX 3090 and 19.6 times faster than the Mac mini. There was a wait of about 2.3 seconds before the first text arrived, but after that the stream exceeded 100 tokens per second, so the difference became increasingly noticeable for longer outputs.

This is a point-in-time measurement that can vary with cloud load and network routing, not a fixed performance specification for the model. Even with that caveat, Luna was not merely a cheaper but slower alternative: under these conditions, it was both the cheapest and fastest option. The two measurements cost approximately $0.005 in total, or about 0.8 yen at 159.3 yen per US dollar.

Build Costs as of August 2026

Due to significant price increases, instead of the price at the time of purchase, the cost to secure an equivalent environment as of August 28, 2026, was estimated. Rather than listing prices, recent closed transactions on Yahoo! Auctions / Furema were used as the baseline as much as possible.

M4 Mac mini 32GB

The 32GB / 256GB configuration this time was 154,800 yen at release, but recent closed transactions for unused / certified refurbished units are 208,000–232,000 yen, with the center rising to around 220,000–230,000 yen. The range is wide for the entire 32GB model due to storage capacity and inclusion of M4 Pro, but the average closed transaction price over the past 180 days was 216,391 yen.

Therefore, the cost to secure the same 32GB / 256GB unit now is set at approx. 220,000–230,000 yen. Additional costs are incurred if an external SSD, keyboard, or display is needed.

RTX 3090 Custom PC

The average for 26 individual closed transactions over the past 180 days for completed used PCs with RTX 3090 + 64GB memory was 242,283 yen. However, the specific examples displayed are mainly transactions from February to June 2026, which is old to use as-is for the reprocurement price at the end of August. The actual examples are also centered on the Core i9-10900K–11900K generation, which is older than the Core i7-13700KF this time.

At current used specialty stores, custom machines with Core i9-12900K, 64GB, RTX 3090, and 1TB SSD are listed at 340,980 yen, and OMENs with Core i9-10850K, 64GB, RTX 3090, 1TB SSD + 2TB HDD are listed at 248,000 yen, both of which were out of stock. Considering CPU generation, 2TB SSD, motherboard, and warranty availability, I think it is reasonable to view the current market price to buy back this configuration immediately as a completed used unit at approx. 340,000–400,000 yen. 240,000–300,000 yen is the price range for waiting for cheap units in individual transactions without warranty, and it is better not to make it a normal reprocurement budget.

New units are even more difficult. This is because the RTX 3090 is not a current product, and what is circulating is a small amount of dead stock. New units as of August 2026 are approx. 200,000–320,000 yen, and the most recent Amazon stock example was 294,840 yen. Adding i7-13700KF, 64GB DDR4, B760M motherboard, 2TB SSD, 1000W power supply, case, and CPU cooler, the reprocurement price to gather same-generation parts from new stock becomes roughly 380,000–480,000 yen.

Buying current products with 24GB or more VRAM as new with warranty is even higher. BTOs with RTX PRO 4000 Blackwell 24GB, 64GB RAM, and 2TB SSD are from approx. 697,800 yen, and machines equipped with RTX 5090 32GB are from approx. 819,800 yen. If purely to run Qwen3.8-27B, it becomes a considerable over-investment.

Comparing with Luna Usage Costs

Luna's API price is $0.20 per 1 million input tokens and $1.20 per 1 million output tokens. Converted at approximately 159.3 yen per dollar on August 27, 2026, 1 million input tokens is approximately 32 yen, and 1 million output tokens is approximately 191 yen.

If 1 million input + 1 million output tokens is considered one set, it is approximately 223 yen. Simply dividing by the hardware price, it takes approximately 1 billion input and output tokens each to reach the 225,000 yen Mac mini, and approximately 1.6 billion tokens each for the 360,000 yen RTX 3090 used machine. If electricity costs, failure risks, and build time on the local side are included, the break-even point moves even further away. Tool costs such as Web search are separate, but cases reaching this order of magnitude in personal use are rare.

Of course, if using a machine already owned, the initial cost is zero, so the story changes. Also, there is value that cannot be measured by money alone for data that cannot be sent externally, offline requirements, and environments that do not depend on API outages.

Is There Meaning in Using a Mac mini?

After knowing 40 tokens/sec, 5.6 tokens/sec feels quite slow. If prioritizing the tempo of dialogue, the RTX 3090 is clearly more comfortable. In actual 2,048-token generation, the difference was about 50 seconds versus about 6 minutes 16 seconds. For long responses, code generation, and iterative execution of agents, this difference accumulates.

Even so, the Mac mini has other strengths.

• Can straightforwardly load a 27B Q4 model into 32GB of shared memory
• Configuration becomes simple with just Metal-compatible llama.cpp
• Completed on a quiet, space-saving local machine
• Can be plugged into existing tools only when needed as an OpenAI-compatible API
• Can be made into a local-first AI environment that does not send input to external APIs

As for my usage distinction, the Mac mini is suitable for "trying locally, handling sensitive text, and launching on-demand at low frequency." The RTX 3090 machine is suitable for "batch processing long-form generation or iterative processing that requires speed." Just because the same model runs does not mean the experience is the same.

However, the conclusion changes for those buying equipment from now on. If there is no interest in the act of building a local LLM itself, no data that cannot be sent externally, and no offline requirements, it is more rational to use Luna first. You can use a model in a similar capability band and a completed tool environment immediately without an initial investment of several hundred thousand yen. A local LLM is closer to reality if considered as a hobby, research, or infrastructure for owning your own computational resources and data paths, rather than a product to save API fees.

Summary

The Qwen3.8-27B Q4_K_M class could be launched on a 32GB M4 Mac mini up to a 128K context setting. Memory pressure remained within the green range, allowing normal screen operations to be performed concurrently, but the generation speed was approximately 5.6 tokens/sec. It maintained 5.44 tokens/sec even in 2,048-token long-form generation, but the required time was 6 minutes 16 seconds. The RTX 3090 machine completed the same output in 49.6 seconds at 41.30 tokens/sec, with a difference of approximately 7.6x.

The most practical view gained from this comparison is to think in the following two stages.

1. Check with capacity whether the model and KV cache fit
2. Estimate the generation speed of the loaded model with bandwidth

Unified memory makes it easier to handle large models, but it is not synonymous with high-speed VRAM of a dedicated GPU. It is true that 27B "runs" on the Mac mini, and it is simultaneously true that the RTX 3090 is significantly "faster." In choosing hardware for local LLMs, it is necessary to consider these two separately.

And, in the decision to purchase new, one more stage is added. At the market price in August 2026, even a Mac mini 32GB is in the 220,000 yen range, and buying back an equivalent RTX 3090 completed used machine through normal sales channels costs approximately 340,000–400,000 yen. Luna's usage-based fee is incomparably small compared to that. If there is no specific motivation to build a local LLM, Luna is fine — this is the blunt conclusion gained from actually building on two machines this time.

The overall picture of related terms is organized in the Local LLM Inference Stack Vocabulary Collection.

References

• Qwen/Qwen3.8-27B
• GPT-5.6 Luna
• GPT-5.4 mini and nano
• chibiham/nix-config