How to Connect to BigQuery from Python

Connecting to BigQuery is unlike connecting to PostgreSQL or MySQL: there is no host, port, or connection string. BigQuery is an API, and "connecting" means authenticating with Google Cloud credentials and pointing a client at a project. That makes the auth setup the actual hard part -- the query code is three lines.

Three libraries matter, all installable from PyPI. Versions verified June 2026.

Which library should you use?

Short version: use google-cloud-bigquery as the default. It wraps the full API and everything else builds on it. Reach for pandas-gbq if your whole interaction is "run SQL, get DataFrame" and you want the one-liner. Use bigframes when the data is too big to pull client-side -- it pushes pandas-style operations down into BigQuery slots instead of copying data to your machine.

Step 0: authentication

Every option below uses the same credential resolution, called Application Default Credentials (ADC). The client looks for credentials in this order:

1. The GOOGLE_APPLICATION_CREDENTIALS environment variable pointing at a service account JSON key file
2. Credentials you set up with gcloud auth application-default login
3. The built-in service account, if running on GCP (Cloud Run, GCE, Cloud Functions)

For local development, the simplest correct setup is:

gcloud auth application-default login

For servers and CI outside Google Cloud, create a service account with the BigQuery Job User role (plus BigQuery Data Viewer on the datasets it reads), download a JSON key, and point the env var at it:

export GOOGLE_APPLICATION_CREDENTIALS="/path/to/key.json"

Treat that JSON file like a password: keep it out of git, rotate it, and prefer workload identity federation over long-lived keys when your platform supports it. Inside Google Cloud, do not use key files at all -- attach a service account to the workload and ADC picks it up automatically.

google-cloud-bigquery (recommended)

Install:

pip install google-cloud-bigquery

Requires Python 3.9+. Connect and query:

from google.cloud import bigquery
 
client = bigquery.Client(project="your-project-id")  # project optional if ADC carries a default
 
rows = client.query_and_wait(
    """
    SELECT name, SUM(number) AS total
    FROM `bigquery-public-data.usa_names.usa_1910_2013`
    WHERE state = 'TX'
    GROUP BY name
    ORDER BY total DESC
    LIMIT 10
    """
)
 
for row in rows:
    print(row.name, row.total)  # rows support attribute and dict-style access

query_and_wait() submits the query and blocks until results are ready. The older pattern you will see in most tutorials -- client.query(sql) followed by .result() -- still works and is what you want when you need the job object itself (for job IDs, statistics, or fire-and-forget jobs).

Note the project in the client is the billing project (where the query job runs); the project in the table reference is where the data lives. That is how you can query bigquery-public-data tables while the bill lands on your project.

Query parameters

BigQuery uses named @parameters, passed via job config -- never f-strings:

from google.cloud import bigquery
 
job_config = bigquery.QueryJobConfig(
    query_parameters=[
        bigquery.ScalarQueryParameter("state", "STRING", "TX"),
        bigquery.ScalarQueryParameter("min_total", "INT64", 1000),
    ]
)
 
rows = client.query_and_wait(
    "SELECT name FROM `bigquery-public-data.usa_names.usa_1910_2013` WHERE state = @state AND number > @min_total",
    job_config=job_config,
)

Results as a DataFrame

pip install 'google-cloud-bigquery[bqstorage,pandas]'

df = client.query_and_wait(sql).to_dataframe()

Two things hide in that extras bracket. pandas pulls in db-dtypes, which maps BigQuery types (DATE, TIME, NUMERIC) onto pandas -- without it, to_dataframe() raises an error telling you to install it. bqstorage installs the BigQuery Storage API client, which downloads results over a parallel gRPC stream instead of paging through the REST API. For results beyond a few tens of MB the difference is large; the client uses it automatically when installed (the service account also needs bigquery.readsessions.create, included in the BigQuery Read Session User role).

Cost controls

You pay per byte scanned (on-demand pricing), and a careless SELECT * over a large table costs real money. Two mechanisms worth wiring in from day one:

# Dry run: how much would this scan?
job_config = bigquery.QueryJobConfig(dry_run=True, use_query_cache=False)
job = client.query(sql, job_config=job_config)
print(f"Would process {job.total_bytes_processed / 1e9:.2f} GB")
 
# Hard cap: fail instead of scanning more than 1 GB
job_config = bigquery.QueryJobConfig(maximum_bytes_billed=10**9)

maximum_bytes_billed fails the query before it runs if the estimate exceeds the cap -- cheap insurance on anything user-facing.

Loading data

df_new = ...  # a pandas DataFrame
job = client.load_table_from_dataframe(df_new, "your-project.dataset.table")
job.result()

For files, load_table_from_file and load_table_from_uri (GCS) handle CSV, JSON, Parquet, and Avro. Loading is free (it uses the shared load pool); streaming inserts via insert_rows_json are not -- prefer batch loads unless you genuinely need rows visible within seconds.

pandas-gbq

pip install pandas-gbq

import pandas_gbq
 
df = pandas_gbq.read_gbq(
    "SELECT name, SUM(number) AS total FROM `bigquery-public-data.usa_names.usa_1910_2013` GROUP BY name",
    project_id="your-project-id",
)
 
pandas_gbq.to_gbq(df, "dataset.table", project_id="your-project-id", if_exists="replace")

Same ADC auth, same Storage API speedup when installed. Use it when the wrapper is all you need; drop down to google-cloud-bigquery the moment you need job configs, parameters, or cost caps -- pandas-gbq accepts a raw API configuration dict, but at that point the core library is clearer.

BigQuery DataFrames (bigframes)

pip install bigframes

import bigframes.pandas as bpd
 
bpd.options.bigquery.project = "your-project-id"
 
df = bpd.read_gbq("bigquery-public-data.usa_names.usa_1910_2013")
result = df.groupby("name")["number"].sum().sort_values(ascending=False).head(10)
print(result.to_pandas())

The pandas-looking operations compile to SQL and execute in BigQuery; data only moves when you call to_pandas(). This is the right tool when the table is 500 GB and your laptop is not. The API covers a large subset of pandas (and scikit-learn, via bigframes.ml) but not all of it, and every operation is a query that bills like one -- the dry-run discipline above applies double.

SQLAlchemy

sqlalchemy-bigquery (1.17.0, maintained by Google) provides a dialect with bigquery://project-id URLs. It works for SQLAlchemy Core queries and tools that speak SQLAlchemy (Superset, etc.). Know its place: BigQuery has no primary keys to enforce and DML is metered differently from OLTP databases, so ORM-style row manipulation is the wrong pattern even though it technically functions.

Common errors

The location one deserves a note: a query job runs in one location (US, EU, a region), and every dataset it touches must live there. Cross-location queries fail with a misleading "not found" error. Check dataset locations before assuming a permissions problem.

Verifying your connection

client.query_and_wait("SELECT SESSION_USER()") confirms which identity you are authenticated as -- the answer is surprising often enough to make this the first debugging step. For browsing datasets, checking table schemas, and estimating scan sizes before you commit to a query, a GUI is faster than the API; see our BigQuery GUI client guide for options. For getting data in, the CSV import guide covers schema autodetection and its failure modes.

Mako connects to BigQuery with AI-powered autocomplete. Try it free at mako.ai.