Thoughts

Brittleness in Software: Meeting in the middle

Software has a unique place in engineering in evolving significantly in design during its construction (often while people are actively using it). Its abstractness, too, makes it hard to get a real sense of what effect changes have on schedules or further work; there is nothing tangible to have our instincts work on.

The issue rears its head particularly when discussing feature plans: there are features that, throughout development, require similar amounts of work to make and some that require orders of magnitude more effort when added later versus being planned in from the start. This huge divide in cost between seemingly similar feature sizes can often challenge recipients with seemingly arbitrarily different estimates. These problems are only exacerbated before an architecture has been decided upon. Continue reading →

Standard
Python

Sharing memory across Python processes using numpy arrays

In a discussion about sharing memory between Python processes in a scientific computing context, several people expressed an interest in a short blog post:

The GIL stops threading being effective for CPU bound tasks in Python and so often-times, people use multiprocessing, an approach which unfortunately heavily restricts the ease of programming (for instance no functions or non-picklable objects can be passed), and the overheads of duplicating data (i.e. in a message queue) can often be too much for large datasets.

Continue reading →

Standard
Startup Thoughts

Why you need a techie from the get go

In the process of making a technology product, as you start to crystallise the idea around what has to be built, there’s going to be a lot of change in the idea, lots of new features are going to come up and no doubt, the “essential” list is going to grow and grow.

Although I’m definitely not against features (and a bit of dreaming), they are definitely the enemy of budget and prioritisation; there is a reason why many experts evangelise the MVP (minimum viable product), mostly because of the ease of spiralling off into an overly complicated product.

While a big list of desired features is inevitable and more features means more work, some features are a lot more work than others, to the point where, once the MVP is specified, huge amounts can be gained just by focusing on easy features over initial desires. Obviously one is going ot have to tackle some difficult features at some point, but one can weigh the two factors together to deliver maximal value within the timeframe. Unfortunately, it requires a certain technical intuition and experience to make those judgements, making it necessary to bring a techie onto the team.

If your idea requires computer intelligence, a well designed architecture is important: how the data is to be represented, how results are calculated or ranked or what specialist techniques will be effective. The problem lies in creating the mental model of the product: without a near full understanding of the required behaviour an architect can often be left playing catchup as new requirements are thrown in after they’ve settled on a design in their head, as their view of the concept changes while they’re required to be producing a design. This can lead to a poor abstraction or possibly a big rethink in the design.

Key to this is communicating the idea well: when inducting a new team member, it is imperative that you communicate the whole vision and concept to them, rather than explaining just the bits you think are relevant: they have to be on the same page. Unless you have some unusual skill in covering everything, getting people in early and often on concept discussions will help keep everyone in sync.

I was recently at a development company (as a client) and was impressed at their structured way of working through every screen and feature to efficiently make concrete the still slightly fuzzy concept. Although this worked well as we had a great idea of what we wanted already, and this pushing us to decisions may well have pushed us into making calls we weren’t ready to make. Collaboration, rather than inquisition, is the order of the day.

Having everyone conceptually in unison will keep everyone working in the same direction and pitching for the same idea. For an effective team everyone needs to own the idea, or they’ll be going after their own, personal, idea.

Standard