How to Connect to BigQuery from Go

BigQuery is not a server you open a TCP connection to. There is no host, no port, no connection string. It is a REST API behind Google's auth layer, and the Go client (cloud.google.com/go/bigquery) wraps that API in a typed client. The two things that actually trip people up are authentication (which credential is the library picking up?) and the dataset location (a query that "can't find" a table is usually looking in the wrong region). This guide covers both, plus parameterized queries, result scanning, and the cost controls you want before you point this at production data.

The package

For querying and ordinary inserts, the main bigquery package is all you need. The Storage Write API is a separate, lower-level path for ingestion pipelines pushing large volumes of rows; most applications never touch it.

go get cloud.google.com/go/bigquery

Authentication: the ADC ladder

The client uses Application Default Credentials (ADC). You do not pass keys in code. The library walks a fixed search order and uses the first credential it finds:

1. GOOGLE_APPLICATION_CREDENTIALS pointing at a service-account JSON key file.
2. Credentials from gcloud auth application-default login (your user account, for local development).
3. The service account attached to the compute resource (Cloud Run, GKE, GCE, Cloud Functions) when running on Google Cloud.

The recommended setup:

• On Google Cloud: attach a service account to the resource and grant it the right BigQuery roles. No keys, nothing to rotate or leak. This is option 3.
• Local development: run gcloud auth application-default login once. This is option 2.
• Outside Google Cloud (another cloud, on-prem): a service-account key file via the env var, treated as a secret. This is option 1, and the one to avoid if you have any alternative, because key files are the thing that ends up in a Git history.

package main
 
import (
	"context"
	"log"
 
	"cloud.google.com/go/bigquery"
)
 
func main() {
	ctx := context.Background()
 
	// projectID is the billing/compute project. It can differ from the
	// project that owns the data you query (see the two-project note below).
	client, err := bigquery.NewClient(ctx, "my-billing-project")
	if err != nil {
		log.Fatalf("bigquery.NewClient: %v", err)
	}
	defer client.Close()
 
	// client is safe for concurrent use. Create it once and share it.
	_ = client
}

The client holds connections and an HTTP transport, so create one at startup and reuse it across goroutines rather than constructing one per request.

Running a query

client.Query builds a query job; Read starts it and returns a RowIterator. Scan rows into a struct whose exported fields match the column names (case-insensitively), or into a []bigquery.Value for ad-hoc shapes.

package main
 
import (
	"context"
	"errors"
	"fmt"
	"log"
 
	"cloud.google.com/go/bigquery"
	"google.golang.org/api/iterator"
)
 
type Customer struct {
	ID     int64
	Name   string
	Region string
}
 
func run(ctx context.Context, client *bigquery.Client) error {
	q := client.Query(`
		SELECT id, name, region
		FROM ` + "`my-data-project.sales.customers`" + `
		WHERE region = @region
		LIMIT 100
	`)
	q.Parameters = []bigquery.QueryParameter{
		{Name: "region", Value: "EMEA"},
	}
 
	it, err := q.Read(ctx)
	if err != nil {
		return fmt.Errorf("query: %w", err)
	}
 
	for {
		var c Customer
		err := it.Next(&c)
		if errors.Is(err, iterator.Done) {
			break
		}
		if err != nil {
			return fmt.Errorf("iterate: %w", err)
		}
		fmt.Printf("%d: %s (%s)\n", c.ID, c.Name, c.Region)
	}
	return nil
}

A few things worth knowing:

• Use named parameters (@region) with bigquery.QueryParameter. Never string-concatenate user input into the SQL; parameters are both safer and let BigQuery cache the query plan. Positional ? parameters are also supported if you set them in order.
• iterator.Done is the sentinel that ends the loop. It comes from google.golang.org/api/iterator, a separate import. Comparing against it with errors.Is is the idiomatic check.
• Struct scanning matches field names to columns, so capitalize your Go fields (ID, not Id, will both match a column named id). For columns that do not map cleanly, use a bigquery.ValueLoader or scan into []bigquery.Value.

The location trap

This is the single most common BigQuery error that looks like something else. Datasets live in a specific location (region or multi-region, like EU, US, or asia-northeast1). If your query job runs in a different location than the dataset it reads, BigQuery returns a "Not found: Dataset ..." error that reads exactly like a permissions or typo problem. It is neither.

Set the location on the query when your data is not in the default (US) multi-region:

q := client.Query("SELECT ...")
q.Location = "EU"
it, err := q.Read(ctx)

If you store everything in one location, set client.Location = "EU" once after creating the client and forget about it. The symptom to remember: a table you can see in the console but the query says "Not found" means check the location before you check anything else.

Cost controls

BigQuery charges by bytes scanned. A single careless SELECT * over a large table can cost real money, and from Go you cannot see the query editor's byte estimate. Two controls matter:

q := client.Query("SELECT ...")
 
// Hard cap: the job fails instead of running if it would scan more
// than this many bytes. 1e9 = ~1 GB.
q.MaxBytesBilled = 1_000_000_000
 
// Dry run: validates the query and reports bytes that WOULD be
// processed, without running it or incurring cost.
q.DryRun = true
job, err := q.Run(ctx)
if err != nil {
	log.Fatalf("dry run: %v", err)
}
status := job.LastStatus()
fmt.Printf("would process %d bytes\n", status.Statistics.TotalBytesProcessed)

Set MaxBytesBilled on any query that runs against untrusted input or large tables. It is the equivalent of a circuit breaker, and it has saved teams from five-figure surprise bills. Use DryRun in tests or pre-flight checks to catch expensive queries before they ship.

Streaming large result sets

The RowIterator pages through results automatically, fetching the next page as you iterate, so reading a million rows does not buffer them all in memory at once. You do not need a separate streaming API for reads. For very large extractions where even paged REST reads are a bottleneck, the BigQuery Storage Read API (cloud.google.com/go/bigquery/storage) provides a faster columnar path, but the standard iterator is correct for the overwhelming majority of cases.

Billing project vs data project

Two project IDs show up and they are not always the same:

• The billing/compute project is the one you pass to bigquery.NewClient. Query costs are charged here, and your credentials need the bigquery.jobs.create permission (the BigQuery Job User role) on it.
• The data project is whatever appears in the fully-qualified table name (`my-data-project.sales.customers`). Your credentials need read access to that dataset (BigQuery Data Viewer is enough for reads).

Querying data in someone else's project while paying for it from yours is normal and expected; just make sure the credential has both the job-creation permission on the billing project and the data-access permission on the data project.

Common errors

Related guides

• Importing CSV data into BigQuery
• The best BigQuery GUI clients
• Connecting to BigQuery from Python
• Connecting to BigQuery from Node.js

Mako connects to BigQuery, ClickHouse, PostgreSQL, and more, with AI-powered autocomplete for exploring your tables. Try it free at mako.ai.