0 comments I regularly run into the need to use part of a URL inside of shell scripts- such as extracting the hostname and port from a URL in order to check if the service is reachable- and got a bit tired of screwing with regex. The urllib python library’s parse component is a great tool for managing this, so I wrote urlparser to expose that library directly from the command line and tossed it up on pypi for others to use.
0 comments When working with the AWS EC2 service in a programmatic way I’ve repeatedly run into a simple problem- how can I get up to date metadata about the various instance types in a programmatic way?
It turns out this simple problem does not actually have a simple solution. AWS offers their Bulk API, which has all the information about every EC2 instance offering in a single giant JSON file, but parsing it with python3.6 will give an OOM error on machines with only 2gb of ram and actually getting the desired data out of it is not a trivial task. The AWS Query API requires AWS credentials and specific IAM roles (and has almost no documentation), making it overly burdensome to use.
Despite that I’ve built-in support for the AWS Bulk API into at least two projects. While contemplating doing it for the third I decided it made more sense to simply build a better API for EC2 Instance Details, with a few goals in mind-
In the end I built a “static” API hosted on GitHub Pages. Every six hours CircleCI kicks off a job to download and process the Bulk API data, generating two files (JSON and YAML) with a cleaned up version of the instance data indexed by instance type. If the files are different from what is already stored in git then CircleCI commits the new files and pushes them back up to GitHub, so the API is never more than six hours out of date from the information available from AWS. Using Github Pages has some real benefits as well, with built-in SSL and the Fastly CDN. The whole system requires no direct hosting on my behalf, and will stay up to date without any need for me to interfere as long as AWS does not change the format of their giant json file. Since the whole thing is stored in git it also creates historical data as a matter of course, showing exactly when changes have occured.
The whole project is, of course, available on Github. The API itself, with documentation, is on Github Pages.
0 comments Last year I introduced a way to manage open source projects with GitConsensus, an open source Github bot that anyone can download and run.
Today running GitConsensus is even easier with the availability of a new GitConsensus GitHub App. Developers can now add GitConsensus directly to their repositories simply by enabling it in GitHub and then adding the consensus files. There’s even example consensus rules covering a variety of options, from anarchy to oligarchy (with a set of recommended rules that is good for most projects).
In order to get this all working I also had to build a new GitHub Apps Python Library. This library extends the excellent github3.py library, making it easier to extend projects already built using it into full-fledged GitHub Apps.
0 comments This weekend I dug into the Github API to build gitconsensus, which lets communities create truly democratic projects using Reactions as a voting mechanism. Projects can define consensus rules (minimum age of pull request, quorum for votes, threshold needed for passing) using a yaml file in their project root. Pull Requests that meet the consensus rules will get merged, and those that do not meet them after a certain amount of time will get closed.
The yaml file itself is pretty straightforward-
# .gitconsensus.yaml # Add extra labels for the vote counts and age when merging extra_labels: false # Do not count any vote from a user who votes for multiple options prevent_doubles: true # Minimum number of voters quorum: 5 # Required percentage of yes votes (ignoring abstentions) threshold: 0.65 # Only process votes by contributors contributors_only: false # Only process votes by collaborators collaborators_only: false # When defined only process votes from these github users whitelist: - alice - bob - carol # Number of days after last action (commit or opening the pull request) before issue can be merged mergedelay: 3 # Number of days after last action (commit or opening the pull request) before issue is autoclosed timeout: 30
The project is available now on pypi and can be installed using pip.
0 comments Over the last year I’ve run into some pretty massive JSON files. One recent examples is from AWS, which publishes a 120mb file containing a list of their available services that they have yet to provide documentation for. Attempting to open that in a standard editor is not going to be pleasant, and while tearing it apart with something like jq is certainly an option it doesn’t feel like the best approach.
That’s why I’ve build jsonsmash, an emulated shell that lets users browse through their massive json files as if they were actually filesystems. It uses shell commands any linux user would already be familiar with to make it very straightforward to figure out the schema of a json object (even if it is a few hundred megabytes) and pull out specific data.
Development is on github and the package is published on npmjs.
0 comments Release v0.14.1 is a major update to Stash, and quite likely the last line of releases before v1.0.0.
The biggest addition is support for PSR-6. Stash now implements the various interfaces natively, allowing it to be directly injected into PSR-6 compliant libraries.
This did require a few API changes (which prompted some cleanup of deprecated functions), so please make sure to review the release notes.
Stash remains one of the best tested caching libraries out there, and this improved on that significantly. In addition there have been optimizations and improvements in PHP7 support, including behind the scenes switching between the APC and APCu functions.
The Symfony Stash Bundle has also been updated, with version v0.6.1 utilizing the Stash v0.14 line.
0 comments 0 comments One of my favorite backup tools has always been rsnapshot. It’s based off of rsync and uses a nice trick with hardlinks to maintain incremental updates that are also full updates. It runs using a basic configuration and a series of cron jobs. This is unix as it’s meant to be- extremely light weight while also being very powerful.
I am rather picky with how it is set up though. I don’t like leaving root open over ssh, which means a sudo based solution is needed on the client side. I’m also rather paranoid, which means I like my backup solutions to be read only. I also don’t like all of my machines running off of a single rsnapshot configuration, as this means a failure for the script to run on one means it won’t run on the ones after.
For years I had a set of scripts to handle this, but in the days of configuration management that seems almost silly. To make life easier I’ve put this all in a Puppet module.
There are quite a few features to this module that make it stand out-
Continue Reading
0 comments The amount and value of data stored on Github Enterprise servers is quite large, and backing them up is rather important. To keep people from doing desperate restores from cloned repositories Github has a tool for backing up and restoring GHE installs. To make this even easier I’ve put together a Puppet module that takes care of the basic setup and configuration of the GHE backups tool.
Combining this module with your existing Puppet environment makes GHE backups easy-
class { 'ghebackups':
ghe_hostname => 'github.example.net',
}
In fact if your network will resolve “github” on your search domains you can even skip the ghe_hostname parameter.
Storing backups with a custom location and number of snapshots is not much more difficult.
file { '/backups/github':
ensure => 'directory'
}->
class { 'ghebackups':
ghe_hostname => 'github.example.net',
ghe_data_dir => '/backups/github',
ghe_num_snapshots => 72,
}
This module also keeps the backup utility up to date and takes care of some other minor issues around installing this package.
For full documentation view the module on Puppet Forge, and as always contributions are always welcome on Github.
0 comments Today I’m announcing the release of Hieratic, a new Puppet module designed to improve the integration between Puppet and Hiera.
Hiera tries to improve Puppet by taking site specific data out of manifests. Hiera makes it possible (and even easy) to define complex sites by making general configurations and then defining only the differences between each node. This can be done on a very granular (and configurable) level. If you haven’t played with Hiera yet take a minute to read through it’s overview and bask in the awesome.
As much as I love it though Hiera is not without flaws. Since it wasn’t shipped by default in Puppet until 3.0 many modules do not have built in support for it. Automatic Parameter Lookup partially solves this problem, but has some serious limitations that significantly reduces it’s usefulness- the biggest being that it is incapable of merging values from multiple levels and only provides the highest priority items.
Hieratic solves this problem. It lets you define a variety of a resources, including classes, while still allowing deep merging to occur. With Hieratic it’s possible to completely separate Puppet code from configuration.
Continue Reading