Integrating LLMs into a Go service without losing your mind (or adding 550ms latency)

Right, so. This is a post I wish existed six months ago when we were first wiring LLMs into our Go backend at Huma. Most of the tutorials out there for LLM integration assume you're in Python. Which is fine — a lot of ML infrastructure is Python, and libraries like LangChain, LiteLLM, and friends are well-documented. But if you're running a Go service stack and you want to add LLM calls without bolting on a whole Python sidecar, the path is less obvious. Here's what we actually learned, including where we went wrong. The problem we were solving We build remote patient monitoring software. Clinicians use dashboards to track patients with chronic conditions — vitals, medication adherence, care notes. We added an LLM-powered summarization layer: given a week's worth of patient data, produce a brief natural-language summary for the clinician at the start of a shift. Simple enough use case. The constraints were: Go service stack (everything is Go at Huma, has been for years) Latency-sensiti