Bye ...!

I've been away many months, most of which I've spent in hospitals. Now, two new heart valves and almost 4 months of heavy antibiotics later I'm starting to get back into 'normal life'. In a few weeks I will hopefully be back doing the thing I do best: create geoscientific software. Or rather, make sure that software gets created and works well.

I've had plenty of time to think about all sorts of things. Things that matter - for me. I have to admit this blog wasn't one of those things. I've dumped my thoughts on many things here, and I got 5 followers (one of which is a son of mine, hi Carlos). I guess it's time to move on. Sorry for those who liked my chattering and ranting: I can't go on. Inspiration and motivation are failing. Maybe some other time.

I'd like to end with the message that something is taking form in my head, something much more related to geoscientific software. If that comes together, I'm pretty sure there will be a new blog. So who knows, maybe ... CU?

Open Source

It's now about 2 months after the previous post. I've been very very busy, and then, last week I finally had a great holiday.


Today I had to clean up some directories, and came across an 'interview'. Magazines send questions and you answer in write, and they distill something from that. Unrecognizable, usually. This wasn't even an interview with me, but I did quite a bit of the answering. The idea was to get a picture of how we coped with Open Source and the business model we chose (it's called freemium). The fun thing is, after just half a year my perspective has changed. Not a lot, but still. Even life itself is an Agile journey.

An excerpt:

"What’s the concept behind open-based platforms?"

Openness in the past has meant one of a few different things. Firstly, there are the almost government-like standardization bodies. They deliver big loads of documents on how things should be done. The problem is that their specifications are hugely complex and there is hardly any help for actual implementation. Next to those there were commercial initiatives that are only open in name only (call them OINO if you like). Commercial interests are huge. And, of course, as soon as any real work needs to be done you have to pay – a lot.

Real openness means you have to open up almost every part of your business. Sharing your products but also your knowledge, data, processes. You share them for a return. The return is manifold, ranging from participation in software development to sharing of data and work flows.

"What is the future of oil & gas software?"

Open Source in the middle, closed-source around that. The Open Source part grows outward, the closed-source parts move further and further into the special-purpose area. Just like in the rest of the software world.

Hmm. Not that sure anymore. In the oil&gas world, Open Source is largely  seen as 'research only'. I went to the EAGE conference in a hopeful, optimistic mode about the workshop on OS. There were actually quite a few people there. But ... well ... the big message we tried to bring was: it also makes business sense. Charles Jones from British Gas gave good evidence for this, too. Actually, he showed that it was even more Agileness that they are after. That made me happy.

But I fear not a lot of other people there were so interested. I mean other than the people there to keep track of sales opportunities. I expected a big positive reaction on Charles' talk - a large oil company interested and really spending money to forward Open Source ...

The crowd just seemed not very interested. My guess is because they were 90% academics. Is Open Source not interesting for the Oil&Gas industry? Hopefully, this is just a big mistake and the closed-source people simply don't see what's coming their way. I sincerely hope so.

Dependencies

I have tried hard to get KDevelop going on CentOS 6. And I wasn't the only one - I saw that it also happened to some people who do such things often. Yes, I have some experience with it myself, too - so I tried it anyway. But this time, I couldn't get Cmake to find Qt. I gave up.

When it doesn't work, the cause of failure is usually a problem with the dependencies. Software engineers know this. Pure developers tend to use as many external tools as possible, but software engineers try to minimize the dependencies. The more experienced you get, the less you like dependencies. As long as you get the services of a perfect system that guards all dependencies you're OK. But it breaks down. What kills is not only that you cannot find the right versions, but also (slight) changes in interfaces or implementations.

Moderns systems are riddled with dependencies. On the OS for one - something you cannot avoid. Luckily OS's are managed by others, and they often do that quite well. As an app developer you will have to counter the annoyances of OS changes, but these are usually not so bad. Even better, OS's provide more an more tools that used to be in separate libraries that you had to manage yourself - separate dependencies.

The decision to start using an external tool is a very important decision. Beginning developers often don't care at all and try to find something for every problem they encounter. They don't bother that they're using just a tiny piece of something, and have to take in a huge mount of dependencies. This quickly results in a dependency nightmare. That's why at OpendTect we actively kick out things as soon as we see that the usage less than strongly required. For example, we did that with FFTW (the MIT FFT library). The benefit of a few percent extra performance in the FFT routines did not outweigh the hassle of the dependency. While kicking out you often diagnose your whole situation, and that makes the operation even more beneficial. We based our FFT on the SU way of doing things, and then found some other things we could do smarter. So in the end we lost the annoying dependency, and FFT-ing is faster than ever before.

For Agile, you don't have to use loads of external stuff. You do have to have basic things like GUI building, 3D visualization, zip/crypt, and so forth. It's important to figure out what can satisfy your needs, but remember that one of your needs is minimal dependencies! Very often your bare needs may be satisfied best using a mix of many small packages. But in the end, you can use much less 'luxury' tools and take in packages that have other good properties like light-weight (no fluff), integrated (just one, rather than many dependencies), robust (maybe not so slick), stable user base (will be there tomorrow, and will still support a changing outside world).

And then ... after a few years you discover that you still bet on the wrong horse. But that's life for you. Murphy's law.

Meta-procedures

Procedures may be useful, but what procedures do we need? This is where what I call meta-procedures are for. How do you define a new procedure, how do you change one, how can you drop one? This must be some sort of community process.

In many organizations the meta-procedure would be: procedures are changed by your boss(es). Needless to say that this is not very motivating. That's why we have something like this:

• if you're not OK with a procedure: speak up.
• if there is consensus that you are right, then we can start a discussion on how to change the procedure
• normally, and while the procedure is not changed: comply (even if you disagree)

You can elaborate on each point (like: what is consensus? who will have the last word, etc), but the idea is clear: we use an established procedure until we decide, as a group, to change it. Also, if you are uncomfortable with a procedure, then discuss it. Chances are that either everyone has the same problem, or that others can enlighten you on the true reason for the procedure. In no case just spoil your and others' mood without any chance for them to plea their cause.

Can meta-procedures be changed? Sure. I can imagine that as your team grows you may want to break with the tradition because upscaling simply requires other decision making processes. Going from closed-source to Open Source may induce something like that, too.

Procedures and creativity

I have been working like mad in the past weeks. Couldn't get myself to writing stuff for this blog. But there's a curious thing I found out in the past years that I want to share. It's the paradox that creativity needs good procedures. Creativity thrives in structured, safe environments.

I read about this topic about a year ago, when the outcome of a study on this issue was covered by all newspapers at the time (here in the Netherlands that is). The surprise was that successful creative companies work well-ordered. The surprise was with the journalists and maybe most of the readers, but for me - no surprise at all.

In our software development team, we are constantly looking for better procedures to handle the standard tasks. We constantly search for better ordering of our working environment. This is engrained in our work structure, and we fight chaos everywhere. Still I might say that we make very creative products, we have literally hundreds of ideas still waiting to be implemented.

With the standard things handled in a standard way you can put your attention to where it pays off: in the areas that matter. No wasting of time and brain cycles on stuff that shouldn't take more than the absolute minimum of your time. If we all adhere, then we're a well-oiled machine producing with optimal performance. Another way to look at it is that good procedures give a feeling of security in the place you live in. We live in our software for a large part of the day. You want to feel safe in your home, you want to know where to find what, and what to do in what circumstances.

The weird thing is that the outside world often thinks of us as being the chaotic hackers. OK, we may not be as orderly as accountants, but in terms of structuring our work we are not bad at all. Oh maybe you are. In that case - change! It pays off.

Becoming a writer

You're reading this. That means my language must be - at least - bearable. I'm well aware that it will never be great. This is partly because English is not my native language. Now that will never change. Also, I lack talent. That won't change, either. What can change, and what has changed in the past years, is that I've become much better at producing reasonable stuff. And I'm faster, too.

When I press my team members to start getting good at it, they often say they are not so interested. They want to program, design, test - and so forth. I think that's a big mistake. If you want to become a great software maker, then your communication skills need to be great, too. That involves many many skills, and the ability to create good texts is one of them. It's not the least of them either. And (this must sound as music to Agile ears) it's re-usable! You'll find these skills useful in many parts of your work and life in general.

It is so important that I always press people to get on with it and do it. The complaint that you're not good at it won't hold: you were not good at writing programs either, years ago. How did you manage to become the programmer/designer/[your job here] you are today? Indeed, the usual way. Studying the art, and doing it - a lot.

So don't try to hide when a text is required from someone but take the opportunity to invest in yourself. "You're worth it".

More worries

I've talked about the necessities of someone worrying about the project before. What should be worried about? Well, that the project is a success ...? But what does that mean? The right things have to be made, and in time, with a certain amount of resources.

Finding out what those 'right' things are, that is a big thing - one of the main raison d'être's for Agile methods. What must be made, and what should it look like so to say.

One technique that helps the developer a lot is to imagine you are now the user. Give yourself a certain task and try to fulfill it. In a way, imagine you have a project that needs this new piece of software (that stuff you are now making). Gather the data and start producing results. Find out how hard it is to use your new stuff. That some things can't be done. Some parts are useless, other parts non-existing.

I call this (and I'm sure I'm not the first) putting on the user's cap. That helps you create a decent 'version 0' product. One that will not be pushed aside with disgust (at least, not immediately).

In some cases that's not it. There may be different types of users. Each with their own cap. Your stuff may be used by end-users and system administrators. Geologists and geophysicists. Managers and technical people. And so forth. Some projects call for imagining the needs of many different types of users. That's a lot of caps, and constant switching is often required.

To support all these roles you may be able to add options, or some extra tools. Sometimes, the roles are so different that you have to make separate programs or branches. Of course that will usually not mean you make different internal objects, just that you offer the services in varying ways. This is a big difference with waterfall analysis that already splits up the functionality from the start. That makes it almost impossible to later join the branches. In Object/Service-oriented environments the counterpart of this 'early split' is lurking. It's the tendency to offer everything as you implement it, rather than making tailor-made things for each of your user types.