Go for AI: where it fits, the libraries, and the patterns that work

Why ask about Go at all?

If you open almost any machine-learning tutorial, the code is Python. The models are trained in Python, the notebooks are Python, and the research papers ship Python repositories. So it is fair to ask why anyone would bring Go into an AI project.

Look at what a working AI product contains. The model is one box. Around it sit a public API, authentication, rate limits, queues, a retrieval service, a vector database, tool servers for agents, background embedding jobs and the cluster software that runs it all. Very little of that is numerical code. Most of it is network code: receive a request, call other services, wait, combine, reply.

Think of a restaurant. The chef is the model, and the chef works best with their own knives, which here means Python and PyTorch. But a restaurant also needs a front door, waiters, a booking system and a till. You would not ask the chef to build those. Go is very good at being the waiters and the front door.

This article covers where Go fits, the libraries for each job, three patterns that work, and common mistakes. Everything named here was checked against the project's own repository or documentation at the time of writing (November 2026); treat any version number as a snapshot.

Where Go fits, and where it does not

Where it fits

Serving and gateways. A Go program compiles to a single static binary. You copy one file into a small container image and run it. There is no interpreter, no virtual environment and no dependency resolution at start-up. For a service that must start quickly, scale out and use little memory, that is a real advantage.

Concurrency-heavy glue. AI back ends spend most of their time waiting on model servers, vector searches and third-party APIs. Goroutines are cheap threads managed by the Go runtime, and channels let them pass work safely. Go's standard server runs each request in its own goroutine, so you write plain sequential code and the runtime handles the waiting. The Go team's own blog post on building LLM-powered applications (Eli Bendersky, September 2024) makes the same point: these applications need good support for REST and RPC, concurrency and performance, which is where Go is strong.

Agents and tool servers. An agent is mostly a loop that calls a model, reads which tool it wants, runs the tool and sends back the result. That is orchestration, not maths, and Go handles it well. There is now an official Go SDK for the Model Context Protocol and official Go SDKs from the main model providers.

Infrastructure. Kubernetes, containerd, etcd and Prometheus are all written in Go. If your models run on Kubernetes, operators and controllers around them are easiest to write in Go, because the client libraries are Go first.

Data plumbing. Moving records between gRPC services, Kafka topics and databases is a natural Go job.

Where it does not fit

Training models. Training needs automatic differentiation, GPU kernels, mixed precision, distributed data parallelism and a large library of layers and optimisers. The Python ecosystem has all of that, maintained by large teams. Go has interesting projects, but nothing close in breadth. If your job is to train or fine-tune a model, use Python.

Research and exploration. Notebooks and trying ten ideas in an afternoon are Python's home ground.

Running GPU kernels yourself. When Go runs a neural network, it usually calls a C or C++ library through cgo (ONNX Runtime, llama.cpp, XLA). That adds a native dependency and a cost per call. For large models, a dedicated model server called over the network is almost always better.

The honest summary: Go is rarely where the intelligence lives, but it is often where the system lives.

The ecosystem by job

The tables below list projects I verified at their official repositories or documentation. One line per project says what it does. Where a project is community-maintained rather than vendor-official, the table says so.

Calling model APIs

All four are typed wrappers over HTTP that handle authentication, retries and streaming. Most also support function calling, where the model asks to run one of your tools.

Agents, frameworks and MCP

If you are starting fresh with MCP, prefer the official SDK. Its README includes a version table: v1.7.0 and later support the 2026-07-28 protocol revision as well as earlier ones.

Serving and inference

Note that the heavy model servers themselves, such as vLLM, are Python and C++. Go's role there is usually the client or the gateway in front.

Vectors and retrieval

For a deeper comparison of the stores themselves, see the pages on Milvus, Weaviate and pgvector.

Data processing

Classic ML and numerics

They suit statistics and simple models inside a Go service, not full training.

Tooling and operations

Pattern 1: a Go gateway in front of a model server

This is the most common way Go enters an AI system, and the easiest to justify.

client ──HTTPS──▶ Go gateway ──HTTP──▶ vLLM (Python, GPU)
                    │  auth, quotas, limits
                    │  logging, tracing
                    └─ routing by model or tenant

The model server does one thing: turn tokens into tokens on a GPU. The gateway does everything else. vLLM provides an OpenAI-compatible HTTP server under /v1, and its own documentation warns that its --api-key option does not protect every endpoint and recommends hardening such as a reverse proxy. That is exactly the gateway's job.

The steps are simple:

1. Check the caller's key against your own store, not the model server's.
2. Apply a concurrency limit so a burst of requests does not overwhelm the GPU box.
3. Forward only the paths you intend to expose, and strip headers that should not travel upstream.
4. Stream the response back as it arrives, because users expect tokens to appear one by one.
5. Record latency, status and token counts for each request.

Here is a minimal version using only the standard library. I compiled and ran it against a stub upstream to check the behaviour.

package main

import (
    "log"
    "net/http"
    "net/http/httputil"
    "net/url"
    "os"
)

func main() {
    // The model server, e.g. vLLM's OpenAI-compatible API.
    upstream, err := url.Parse("http://127.0.0.1:8000")
    if err != nil {
        log.Fatal(err)
    }
    proxy := &httputil.ReverseProxy{
        Rewrite: func(r *httputil.ProxyRequest) {
            r.SetURL(upstream)
            r.Out.Header.Del("Authorization") // do not leak client keys upstream
        },
        FlushInterval: -1, // flush immediately so streamed tokens arrive as they are produced
    }

    slots := make(chan struct{}, 64) // at most 64 requests in flight to the GPU box
    apiKey := os.Getenv("GATEWAY_KEY")

    http.HandleFunc("/v1/", func(w http.ResponseWriter, r *http.Request) {
        if r.Header.Get("Authorization") != "Bearer "+apiKey {
            http.Error(w, "unauthorised", http.StatusUnauthorized)
            return
        }
        select {
        case slots <- struct{}{}:
            defer func() { <-slots }()
        default:
            http.Error(w, "busy, retry later", http.StatusTooManyRequests)
            return
        }
        proxy.ServeHTTP(w, r)
    })
    log.Fatal(http.ListenAndServe(":8080", nil))
}

The buffered channel slots works like numbered tickets at a deli counter: take one to be served, hand it back when you leave, and if none is free, come back later. In production you would add a real key store, constant-time comparison, timeouts and tracing, but the shape stays the same.

Pattern 2: a Go agent loop with tools and an MCP server

An agent, stripped down, is a loop. The agentic AI page goes deeper into designs, but the core is short:

1. Send the conversation and the list of available tools to the model.
2. If the reply contains tool calls, run each tool and append the results to the conversation.
3. Go back to step 1.
4. Stop when the model replies with plain text, or when you hit a step limit.

Two practical rules make this safe. First, always cap the number of steps, so a confused model cannot loop forever. Second, give every tool call a context.Context with a deadline, so a slow tool cannot hold the whole request hostage.

Tools can live inside the agent process, but the Model Context Protocol lets you put them in a separate server that any MCP-capable client can use. It is like a standard plug socket: once your tool speaks MCP, any appliance with the right plug can use it.

With the official Go SDK, a tool is a typed Go function. The SDK infers the JSON schema for the input and output from the struct fields and jsonschema tags, validates the arguments and returns structured content. This server exposes one tool over standard input and output. I built it and called it with a raw JSON-RPC initialize and tools/call exchange; the reply contained "structuredContent":{"words":5} for a five-word input.

package main

import (
    "context"
    "log"
    "strings"

    "github.com/modelcontextprotocol/go-sdk/mcp"
)

type CountIn struct {
    Text string `json:"text" jsonschema:"the text to analyse"`
}

type CountOut struct {
    Words int `json:"words" jsonschema:"number of words in the text"`
}

func countWords(ctx context.Context, req *mcp.CallToolRequest, in CountIn) (*mcp.CallToolResult, CountOut, error) {
    return nil, CountOut{Words: len(strings.Fields(in.Text))}, nil
}

func main() {
    server := mcp.NewServer(&mcp.Implementation{Name: "text-tools", Version: "v0.1.0"}, nil)
    mcp.AddTool(server, &mcp.Tool{
        Name:        "count_words",
        Description: "Count the words in a piece of text",
    }, countWords)

    if err := server.Run(context.Background(), &mcp.StdioTransport{}); err != nil {
        log.Fatal(err)
    }
}

For a remote server rather than a local process, the same package provides NewStreamableHTTPHandler, which turns your server into an http.Handler. You then mount it behind the same kind of gateway as in Pattern 1, with authentication in front. Note that recent releases of the SDK require a recent Go toolchain; check its go.mod before you pin a version.

Your agent then becomes an MCP client: it lists the tools, passes their schemas to the model as function definitions and routes each call to the right server. Genkit, Eino and LangChainGo package this loop, but writing it by hand once shows you where errors, timeouts and costs come from.

Pattern 3: a concurrent embedding pipeline

Building a retrieval index means turning many documents into embeddings. The embedding model usually sits behind an API or a model server that accepts batches. Sending one text at a time wastes round trips; sending everything at once hits size limits. The usual answer is batching plus a fixed pool of workers.

texts ─▶ split into batches ─▶ jobs channel ─▶ N workers ─▶ embedder
                                                   │
                         results written back by position ◀┘

The key ideas:

• Batch size controls how much each call carries. Pick what the provider allows and what fits comfortably in memory.
• Worker count controls how many calls are in flight at once. This is your politeness setting towards the provider's rate limits.
• Order matters, because vector i must match text i. Each worker writes into its own slice range, so no locking is needed.
• Cancellation matters, because if one batch fails you usually want the others to stop rather than burn quota.

The function below works with any embedder that satisfies a small interface, so you can plug in a provider SDK, a local Ollama model or a test fake. I ran it with the race detector enabled, with both a working and a failing fake.

package main

import (
    "context"
    "sync"
)

// Embedder is anything that turns a batch of texts into vectors:
// a provider SDK, a local model server, or a test fake.
type Embedder interface {
    Embed(ctx context.Context, texts []string) ([][]float32, error)
}

type job struct {
    start int
    texts []string
}

// EmbedAll splits texts into batches and embeds them with a fixed
// number of workers. The output keeps the input order.
func EmbedAll(ctx context.Context, e Embedder, texts []string, batchSize, workers int) ([][]float32, error) {
    ctx, cancel := context.WithCancel(ctx)
    defer cancel()

    out := make([][]float32, len(texts))
    jobs := make(chan job)
    var wg sync.WaitGroup
    var once sync.Once
    var firstErr error

    for w := 0; w < workers; w++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for j := range jobs {
                vecs, err := e.Embed(ctx, j.texts)
                if err != nil {
                    once.Do(func() { firstErr = err; cancel() })
                    return
                }
                copy(out[j.start:], vecs) // each worker writes its own slice range
            }
        }()
    }

feed:
    for i := 0; i < len(texts); i += batchSize {
        end := min(i+batchSize, len(texts))
        select {
        case jobs <- job{start: i, texts: texts[i:end]}:
        case <-ctx.Done():
            break feed
        }
    }
    close(jobs)
    wg.Wait()

    if firstErr != nil {
        return nil, firstErr
    }
    return out, ctx.Err()
}

Picture a post office with four counters and a queue of parcels. Each clerk takes the next parcel; if one finds a dangerous parcel, they pull the alarm and the queue closes. In a real pipeline you would add retries with back-off for rate-limit errors.

Performance and concurrency, explained simply

Goroutines are cheap. A goroutine starts with a small stack that grows as needed, so a server can hold many thousands of them while they wait on network calls. This is why a Go gateway can keep many slow model requests open at once without a thread per request.

Waiting is free; computing is not. Goroutines help when the work is mostly waiting. They do not make maths faster. If your Go service spends its time on matrix multiplication, more goroutines will not save you; a GPU and a proper kernel will.

Know your CPU budget in containers. Since Go 1.25, on Linux, the runtime sets GOMAXPROCS from the container's cgroup CPU limit when that limit is lower than the machine's CPU count, and updates it if the limit changes. In Kubernetes terms, it follows the CPU limit, not the CPU request. If you set no limit, Go will see every core on the node.

Garbage collection is tunable. Go is garbage-collected. For most AI glue code the pauses are short and unimportant compared with model latency. Go 1.26 made the newer "Green Tea" collector the default. If a service holds large in-memory indexes, measure heap use and tune GOGC or set a memory limit with GOMEMLIMIT rather than guessing.

cgo has a cost. Calling C from Go crosses a boundary each time. The Go 1.26 release notes state that the baseline overhead of cgo calls was reduced by about 30 per cent, but it is still not free. The practical rule: make few, large calls across the boundary (run a whole batch through ONNX Runtime in one call), not many small ones.

SIMD is arriving. Go 1.26 added an experimental simd/archsimd package for amd64 and Go 1.27 an experimental portable simd package, both behind GOEXPERIMENT=simd. Useful for distance maths, but not yet something to build a product on.

Go, Python and Rust for AI work

Python and Rust are the two languages teams most often weigh Go against. Python is the training language; Rust is the other systems language with a strong AI footprint (Hugging Face's tokenizers core and the candle framework are Rust, and so is the Qdrant database).

The short version: Python for the model, Go for the services, Rust when you need maximum control over memory and latency inside a hot component and have the people to maintain it.

A practical decision guide

1. Are you training or fine-tuning? Use Python. Do not look for a Go alternative.
2. Are you calling hosted models from a product back end? Go is a strong choice. Use the official SDK for each provider.
3. Are you serving a large open model on GPUs? Run vLLM, llama.cpp or a similar server, and put a Go gateway in front for auth, limits and routing.
4. Do you need a small model inside a Go service (a classifier, a reranker, a sentence embedder)? Export it to ONNX and run it through ONNX Runtime bindings or Hugot. Keep calls batched.
5. Are you building tools for agents? Write them as MCP servers with the official Go SDK, so any client can use them.
6. Are you building the agent itself? Write the loop by hand once. Then decide whether Genkit, Eino or LangChainGo saves you enough to justify the dependency.
7. Are you building a retrieval pipeline? Go handles ingestion, batching and querying well. Pick the vector store on its merits; Weaviate and Milvus are Go-based, and most others have Go clients.
8. Is your team mostly data scientists? Keep them in Python and put Go only at the edges.

Common mistakes

Rewriting the model in Go. Porting a trained network to a Go numerics library is almost never worth it. Export to a standard format or call a server.

One request, one model call, no limit. Without a concurrency cap, a traffic spike passes straight through to the GPU server and everyone's latency collapses. Always put a limit, a queue or both in front of expensive calls.

Ignoring context cancellation. If a user closes the browser, the request context is cancelled. Pass that context down to every model and tool call, so you stop paying for work nobody will read.

Loose tool schemas. If a tool accepts a free-form string, the model will send surprising things. Use typed input structs, describe each field, and validate before acting. This matters even more when the tool writes data.

Choosing a dormant library. Check the last commit and release before you adopt a Go ML library. Some well-known names have been quiet for years.

Hiding cgo in the build. ONNX Runtime and DuckDB bindings need native libraries. Decide early how they reach your container, or the static-binary advantage disappears.

Frequently asked questions

Can I train a neural network in Go?

You can, with GoMLX or Gorgonia, but you give up most of the tooling and pretrained models that make training practical. Beyond small experiments, train in Python and bring the model to Go through ONNX or a model server.

Is there an official Go SDK for OpenAI, Anthropic and Google models?

Yes: openai-go, anthropic-sdk-go and google.golang.org/genai, each maintained by the provider. Check the current major version before writing import paths, because Go module paths change with major versions.

What is the best way to call a local model from Go?

The simplest is Ollama, itself written in Go, which exposes a REST API on port 11434. Call it with its Go client package or, through its OpenAI-compatible endpoints, with an OpenAI-style client. For GPU serving under heavier load, vLLM is the more common choice.

Should I use the official MCP Go SDK or a community one?

For new work, the official SDK under the modelcontextprotocol organisation is the safer default, because it tracks the specification and publishes a compatibility table. The community mcp-go project has been developed since late 2024 and is still active, so there is no urgent reason to migrate a working server, but new servers are simpler to keep current on the official package.

Is Go fast enough for vector search?

Yes. Weaviate is written in Go, and Milvus is written in Go and C++. For your own service, the vector search itself usually happens in the database, and Go's job is to build the query, call the store and merge the results, which it does well.

Do I need a framework like Genkit, Eino or LangChainGo?

Not to start. A tool-calling loop is a few dozen lines with an official SDK. Frameworks pay off when you want retrievers, prompt management, tracing, graph workflows or a developer UI; pick one once you know which you need.

Summary

Go is not a training language, and it does not try to be. Its strengths are the ones an AI product needs around the model: single-binary deployment, cheap concurrency for workloads that mostly wait, strong networking and gRPC support, and a natural fit with Kubernetes and the rest of the cloud-native stack. Today there are official Go SDKs for the main model APIs and for the Model Context Protocol, several agent frameworks, Go-native vector databases and solid data libraries. The patterns that work are the simple ones: a Go gateway in front of a Python model server, a bounded agent loop with tools exposed over MCP, and a worker pool that batches embedding calls. Keep the model in Python, keep the system in Go, and make every expensive call bounded, cancellable and observable.