feat: (Preview) Support automatic load of timedelta from BQ tables.#1429
feat: (Preview) Support automatic load of timedelta from BQ tables.#1429
Conversation
|
Closing the PR for now to think about annotations for RECORD and ARRAY. I will prepare another PR once everything falls into place |
| self._attach_timedelta_metadata(query_job.destination, array_value) | ||
| return query_job | ||
|
|
||
| def _attach_timedelta_metadata( |
There was a problem hiding this comment.
why not put this logic instead in convert_to_schema_field? it is the natural inverse of convert_schema_field
There was a problem hiding this comment.
Good call! That makes the code cleaner.
There was a problem hiding this comment.
Though interestingly, in export_gbq we rely on the BigQuery engine to figure out the schema based on the provided SQL. I need to update the schema again afterwards to include the tag
| elif ( | ||
| field.field_type == "INTEGER" | ||
| and field.description is not None | ||
| and TIMEDELTA_DESCRIPTION_TAG in field.description |
There was a problem hiding this comment.
nit: but I do wonder if once in a while, we might catch a user description that accidentally uses '#microseconds' somewhere? maybe restrict to beginning, or put some odds character sequence to ensure non-accidental triggering
There was a problem hiding this comment.
SG. Changed the logic to check whether the description ends with the tag.
This works only with
read_gbq_table()under the hood, becauseread_gbq_query()erases column descriptions.