Skip to content

Coerce BigQuery types when inserting #129

@poundifdef

Description

@poundifdef

The following inserts into bigquery will fail:

{"x": "y"}
{"x": true}
googleapi: Error 400: Invalid schema update. Field x has changed type from STRING to BOOLEAN, invalid

This is because when the table is first created, column x is a string. When the second JSON is inserted, x is a bool. We want to try and coerce data to match the column type if possible, otherwise use a null value. This will prevent inserts from failing outright.

BigQuery lists conversion rules here which may be helpful: https://cloud.google.com/bigquery/docs/reference/standard-sql/conversion_rules

  • Create new configuration option big BigQuery: strict_types bool. If it is set to true, then do not do any type conversion. If it is false, then do some sort of type conversion (either in SQL or on the go side) when inserting data.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions