BigQuery exporting - possible? #12246
andrewm4894
started this conversation in
Ideas
Replies: 1 comment
-
@vlvkobal please take a look |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
It would be very useful to implement a BigQuery exporting engine such that users could just leverage BQ for long term storage and analytics.
We actually need to build something like this internally for our prod nodes to serve as the background for proper long term storage to support ML research efforts easily on our own production data.
Currently we have a Pub/Sub exporter but its still a bit too complicated to go from Pub/Sub to BQ and involved users having to set up dataflow jobs or some other ETL. It would be much better instead to just stream the data direct to BigQuery.
BigQuery streaming api: https://cloud.google.com/bigquery/streaming-data-into-bigquery
BigQuery Go client library: https://cloud.google.com/bigquery/docs/reference/libraries#client-libraries-usage-go
Would something like this be feasible to build?
Beta Was this translation helpful? Give feedback.
All reactions