I have been experimenting with using google Bigquery recently.

It looks to be an incredibly useful (and cheap) means of storing and using lots of data.

In fact it is so cheap that asking management for permission in a meeting can cost more in staff time than a years use of the data. This means that with the right policy it is possible to store whatever data you think might be of use.

The downside is the documentation. It looks comprehensive at first until you try to use it for something. The java code samples are full of sections such as:

// Insert useful code here

There does not even seem to be a comprehensive list of the datatypes so here goes:

STRING

BOOLEAN

TIMESTAMP

INTEGER

FLOAT

These fill in for where there should be key points (such as defining the fields in a Bigquery table).

Bigquery does have a sensible structure:

Projects are for billing.

These contain datasets that are used for access control (we have a set of these per environment : qa, demo).

Datasets contain tables (and views – but I have not used these yet).