:::: MENU ::::
Browsing posts in: Open Source

Introducing jsonsmash – work with large json files easily

Over the last year I’ve run into some pretty massive JSON files. One recent examples is from AWS, which publishes a 120mb file containing a list of their available services that they have yet to provide documentation for. Attempting to open that in a standard editor is not going to be pleasant, and while tearing it apart with something like jq is certainly an option it doesn’t feel like the best approach.

That’s why I’ve build jsonsmash, an emulated shell that lets users browse through their massive json files as if they were actually filesystems. It uses shell commands any linux user would already be familiar with to make it very straightforward to figure out the schema of a json object (even if it is a few hundred megabytes) and pull out specific data.

Development is on github and the package is published on npmjs.


Building an Email Testing Environment with Vagrant, Dovecot and Travis-CI

In my previous post I introduced the testing suite I created for Fetch. Here I want to go through exactly what needed to be done to put that together. This post is a bit longer, and you really don’t need it to take advantage of this package, but it may provide some insight to anyone hoping to do something like this themselves.

For me testing has two major components- it should ease development on the local level by preventing regressions, and it should help me as the maintainer of an open source project by giving me trust in the pull requests that are coming in to me. Travis-CI, an application I’ve been using extensively for my projects now, makes that last problem much easier as it will run my test suites and tell me right in the pull request if any problems have come up. On the local level I decided to use Vagrant, as it makes adding virtual machines to source control (without adding a huge binary file) really simple.

Continue Reading


Announcing a New Continuous Integration and Email Package using Travis-CI and Vagrant

Years ago I wrote a library, Fetch, which was designed to read email using the PHP IMAP Extension. At the time I did not expect it to get much reuse so I skipped out on much of the test suite- if I’m going to be perfectly honest here though I have to admit that a huge reason for not building out a test suite then was that I was unsure how to go about it. Building a test suite for an email library has some very interesting requirements, especially if you want to test it in as real world a setting as possible.

  • You need a mail server.
  • That mail server needs a variety of messages, folders, flags and attachments for all the for different types of tests that are needed.
  • Those messages have to remain consistent between test runs.
  • Resetting the mail server to it’s original state should be fast to encourage lots of testing during development.
  • The test suite, and thus the mail server, need to be portable so other people can run tests on their changes.
  • Continuous integration (integrated in with something like Github) is really really nice.

Unfortunately for me all of my great excuses have vanished, thanks in large part to Travis-CI and Vagrant. Both of these systems are fantastic for testing. Travis is a continuous integration environment that’s setup for a number of languages and has direct integration with a number of services such as Github. Vagrant makes sharing server environments as easy as passing along a configuration file. With the two of these tools I was able to put together a testing package for Fetch that met all of those requirements, and after a bit of additional work I’ve broken that up into a stand alone package for people to use in their own projects.

Using this is really simple- you just need to call SetupEnvironment.sh before each run of your test suite. Once that is finished running you have a fully functional Dovecot server with a consistent set of messages. It doesn’t matter if you’re running this from your home computer or if it’s run on Travis-CI, as the SetupEnvironment.sh script will recognize where it’s being run and work accordingly.

There are also some cool optimizations to make testing faster. The test environment will stay up and running for a half hour after the last test before shutting itself down, so you don’t have to wait for the virtual machine to boot between tests. Rather than start from a blank slate each time, the environment has a reset function that restores the original inboxes. With an already provisioned test environment there is almost no overhead by this package between tests, as resetting the inbox takes only a couple of seconds.

The project is up on Github at tedivm\DovecotTesting, so if you’re writing an email reading library there’s no more excuses. I’ve also put together a more detailed post describing how this was put together.


WordPress Syntax Highlighting for YAML

I was writing up a new blog post that had some Yaml config files in them. I’ve been using the Syntaxhighlighter Evolved plugin for this, which has worked remarkably well. Unfortunately it seems that Yaml is not one of the supported languages, and I couldn’t find much about it.

Luckily for me the author wrote a fantastic blog post about extending his plugin, which I used to create my very first WordPress plugin. If you’re looking to add Yaml highlighting to your blog you can grab my SyntaxHighlighter Evolved: Yaml Brush plugin right off of the WordPress Plugin Directory. The first version is pretty simple, but I’m considering collecting a few different languages together for a more comprehensive pack.


PHP IMAP Script

As part of maintaining Fetch I have to install the php imap extension quite a bit. Although this is pretty trivial on most variants of linux it’s kind of a pain for OSX- you have to find a few dependancies, compile the imap c library from source, create the extension against your currently installed version of php (which typically won’t include it’s source on the system), and then take that one resulting file and set it up.

After doing this for the millionth time I decided to script it, and like any good programmer I looked to see what was already out there. I found a script by Ivan Vucica for an older version of OSX and then polished it up. My version should work on more than just OSX, although in most cases you’ll want to use the system package manager in that case anyways.

I’ve posted the code and a Readme on Github, and have thrown together a first release. Please let me know if you find it useful!