databases

If you have a large Postgres database, consider trying Google Cloud BigQuery for OLAP-style reporting queries. database> COPY (<query_to_denomralize_report_data>) TO '<dest_filename>' DELIMITER ',' CSV HEADER; Then upload it to GCS: $ gsutil cp <dest_filename> gs://<dest_bucket>/<dest_filename> Then, follow Loading Data into BigQuery (with Google Cloud Storage). For this to work, you’ll have to define the schema and skip the first leading row. My most recent load job took 4 seconds, loading a very large CSV file....

Some days I feel that I’ve returned to my old DBA role, well… sort of. A lot of my day to day work revolves around a vast amount of data and applications for data storage and retrieval. I haven’t been a DBA by trade since 2011, nearly an eternity in the tech industry. To me, Designing Data-Intensive Applications was a wonderful survey of “What’s changed?” as well as “What’s still useful?...

I’ve been looking hard at datastores recently, in particular PostgreSQL + PostGIS. In school, I did a few GIS related things in class and I assumed it was magic. It turns out there is no magic. At work, we have an application that accepts polygons and returns objects whose location are within that polygon. In GIS-speak this is often called an Area of Interest or AOI and they’re often represented as a few data types: GeoJSON, Well-known Text and Well-known Binary (WKT/WKB)....