I kept seeing this error message with a uwsgi/Flask/SQLAlchemy application: [2019-09-01 23:43:34,378] ERROR in app: Exception on / [GET] Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 2292, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1718, in handle_user_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "./http.py", line 11, in foo session....

If you have a large Postgres database, consider trying Google Cloud BigQuery for OLAP-style reporting queries. database> COPY (<query_to_denomralize_report_data>) TO '<dest_filename>' DELIMITER ',' CSV HEADER; Then upload it to GCS: $ gsutil cp <dest_filename> gs://<dest_bucket>/<dest_filename> Then, follow Loading Data into BigQuery (with Google Cloud Storage). For this to work, you’ll have to define the schema and skip the first leading row. My most recent load job took 4 seconds, loading a very large CSV file....

A quick note: Don’t naively use dateutil.parser.parse() if you know your dates are ISO8601 formatted. Use dateutil.parser.isoparse() instead. In [1]: from dateutil.parser import * In [2]: datestr = '2016-11-10 05:11:03.824266' In [3]: %timeit parse(datestr) The slowest run took 4.78 times longer than the fastest. This could mean that an intermediate result is being cached. 1000 loops, best of 3: 211 µs per loop In [4]: %timeit isoparse(datestr) 100000 loops, best of 3: 17....

One of my goals is lifelong learning. Since finishing school over a decade ago, most of my learning has taken place on the job or through reading technical books in the last decade. I often feel that books provide great surveys and details on theory (which are important!) but they’re light on useful information for practitioners. Coursera Coursera was the first experience I had with online classes. I’ve done three courses on Coursera: Computer Networks ( available here), Algorithms Part I, and Algorithms Part II....

Some days I feel that I’ve returned to my old DBA role, well… sort of. A lot of my day to day work revolves around a vast amount of data and applications for data storage and retrieval. I haven’t been a DBA by trade since 2011, nearly an eternity in the tech industry. To me, Designing Data-Intensive Applications was a wonderful survey of “What’s changed?” as well as “What’s still useful?...

Google Cloud Platform has the notion of resource labels which are often useful in tracking ownership and cost reporting. Google Cloud BigTable supports data exports as sequence files. The process uses Cloud Dataflow. Cloud Dataflow ends up spinning up Compute Engine VMs for this processing, up to –maxNumWorkers. I wanted to see what this was costing to run on a regular basis, but the VMs are ephemeral in nature and unlabelled....

I’ve been looking hard at datastores recently, in particular PostgreSQL + PostGIS. In school, I did a few GIS related things in class and I assumed it was magic. It turns out there is no magic. At work, we have an application that accepts polygons and returns objects whose location are within that polygon. In GIS-speak this is often called an Area of Interest or AOI and they’re often represented as a few data types: GeoJSON, Well-known Text and Well-known Binary (WKT/WKB)....

I had to fight a little with GPG Suite (Mac) and forwarding gpg-agent to a Ubuntu 16.04 target. This post describes what ended up working for me. Source Machine: macOS Sierra 10.12.6 GPG Suite 2017.1 (2002) Target Machine: gpg2 installed on Ubuntu 16.04 (sudo apt-get install gpg2 -y) Source machine: File: ~/.gnupg/gpg-agent.conf Add this lines: extra-socket /Users/[user]/.gnupg/S.gpg-agent.remote File: ~/.ssh/config Add these lines: RemoteForward /home/[user]/.gnupg/S.gpg-agent /Users/[user]/.gnupg/S.gpg-agent.remote ExitOnForwardFailure Yes Restart gpg-agent: gpgconf --kill gpg-agent gpgconf --launch gpg-agent Destination Machine: File: /etc/ssh/sshd_config Add this line:...

I found and upvoted this solution on ServerFault. TLDR: Use OS X Disk Utility to Create a New Sparse, Case Sensitive Disk Configure VirtualBox to use this new disk Some screenshots that may be helpful: From Disk Utility -> File -> New Image: From VirtualBox -> Preferences: 

While debugging an Ansible playbook, I tweeted about variable precedence: https://twitter.com/andyhky/status/734859666337861633 Through that tweet I learned how to spell mnemonic AND I got a recommendation to buy Mastering Ansible. This blog post is a quick review of Mastering Ansible. Overall, I enjoyed the book’s format more than anything else. The flow of objective/code/screenshot really made the concepts gel (using herp/derp in examples is a close second!). I’m an advanced Ansible user (over 3 years), and this book still showed me new parts of Ansible....