2017 comes with the decommissioning of two cloud ci providers.

Snap-CI and Bamboo-CI are both shutting down.

Snap-ci provided a build pipeline (triggered by a github webhook) with log and artefact viewing. This was all configured using a clean UI which allowed environment variables to be stored as either plaintext or securely.

Finding a replacement is not simple. The options seem to include options without a ui to specify the pipelines or are much more expensive.

The roll your own optiona are complex possibly including docker hosted travis.

This open source pivot table (http://nicolas.kruchten.com/pivottable/examples/local.html) brings the power of Lotus Improv to the web.

Lotus Improv was a groundbreaking spreadsheet that was based upon pivot tables rather than a simple grid. Lotus killed it as it was cannibalising it’s own Lotus 1-2-3 sales.

The pivot table above allows you to load a csv file (say the output of a Bigquery query) and then perform ad-hoc pivot analysis on it.  By dragging the column headings around you get to see your data summarised in your browser.  The sample could easily be extended to run a report from say a S3 bucket. This means that we could email links to pivot tables around…

After my last post I have found that Amazon (AWS), Microsoft (Azure) and Google (Google Cloud) are all in the process of constructing UK based regions. However they don’t seem to have gone for the Northern Powerhouse option…

Opsgenie seems to dominate my life for a week every other month.

OK, it’s a support rota.

However it’s a great way of getting teams to do things. Raise it as an alert and it can’t be ignored. It’s a simple http post to raise or close so automation is trivial.

Intgration with statuspage.io as a webhook very is powerful.

This is an unusual kind of a database.

To start with it’s very cheap to use – you generally only pay for the columns of data read. You can get a lot of data processing for a few pence.

It does have some interesting other characteristics:

You can only read and write data, you can’t update it or delete specific rows.

It’s not always very fast (it’s for analytics not transactional  data) and does have periodic outages.

It can contain nested repeated rows.

Tables are meant to be extended – you can add an optional  or repeated column but can’t change the type of an existing one.

Copying tables is effectively free.

You can replace the entire contents of a table with a select result (or create a new table from a select result).

This means that you need a different strategy for loading data into these tables. The trick is to find a means of making inserts idempotent without reading too much data.

This could involve writing to a staging table and using a copy insert to move distinct data back into the master.

You don’t have to use the rest of the google infrastructure to load into BQ – we use heroku scheduler tasks. This means we pay pennies to load data into a storage system that costs pennies to run. This can completely change the economics of software development. The most significant cost is now developer and management time – it can now be cheaper to delegate authority for data creation to the developer (only ask if it will cost more than $100 per month) than to hold a meeting to request approval.

This cloud economics also gets fun when deciding when to optimise a slow process. If you could dial up the size of the box that you use to add 7 pennies to the months bill then how many months would that process need to run in order for it to be worth spending 2 hours of developer time to save that cost?

• A pricing model that scales with our success and allows for spikey traffic.
• A stable interface. If you want to change something make it backwards compatable or give us notice of change.
• Have a status page similar to status.io that is accurately updated.
• Respond to support requests in hours not days.
• Use webhooks rather than emails to announce outages, maintenance windows and upgrades.
• We need to use an api to deploy or configure your service.
• We need an api to reconcile any billing transactions with payments.
• We will have multiple environments (qa, perf, staging, live) and only expect full price and support on live and perf.

Recently I have been working on a data warehouse project that is trying to extract billing and payment information from a range of cloud services. In all cases it seems that the data needed by the accountants has not been given the care that it needs.

The industry practice seems to be log into a portal and download a spreadsheet of data that may be accurate or timely. Any api’s exposing this data seem to only work on a per order basis – which means that an attempt to keep a database upto date could require geometrically increasing processing times (you need to check each order for changes to a transaction).  These api’s tend to be heavily restricted on the volume of data per call (hundreds of transactions per call) combine this with a frequently undocumented rate limit make catching up historical data difficult at best.

If you are providing services that accountants need think about providing an api that makes reconciliation easy. Reconciliation is important especially if you want to get your tax calculations right.

I am getting increasingly frustrated with the quality of the packages in NPM.

Is seems that it’s ok to randomly change code and leave things in a broken state (or at least have a stack of errors for simple usage.