Open Source

It's now about 2 months after the previous post. I've been very very busy, and then, last week I finally had a great holiday.


Today I had to clean up some directories, and came across an 'interview'. Magazines send questions and you answer in write, and they distill something from that. Unrecognizable, usually. This wasn't even an interview with me, but I did quite a bit of the answering. The idea was to get a picture of how we coped with Open Source and the business model we chose (it's called freemium). The fun thing is, after just half a year my perspective has changed. Not a lot, but still. Even life itself is an Agile journey.

An excerpt:

"What’s the concept behind open-based platforms?"

Openness in the past has meant one of a few different things. Firstly, there are the almost government-like standardization bodies. They deliver big loads of documents on how things should be done. The problem is that their specifications are hugely complex and there is hardly any help for actual implementation. Next to those there were commercial initiatives that are only open in name only (call them OINO if you like). Commercial interests are huge. And, of course, as soon as any real work needs to be done you have to pay – a lot.

Real openness means you have to open up almost every part of your business. Sharing your products but also your knowledge, data, processes. You share them for a return. The return is manifold, ranging from participation in software development to sharing of data and work flows.

"What is the future of oil & gas software?"

Open Source in the middle, closed-source around that. The Open Source part grows outward, the closed-source parts move further and further into the special-purpose area. Just like in the rest of the software world.

Hmm. Not that sure anymore. In the oil&gas world, Open Source is largely  seen as 'research only'. I went to the EAGE conference in a hopeful, optimistic mode about the workshop on OS. There were actually quite a few people there. But ... well ... the big message we tried to bring was: it also makes business sense. Charles Jones from British Gas gave good evidence for this, too. Actually, he showed that it was even more Agileness that they are after. That made me happy.

But I fear not a lot of other people there were so interested. I mean other than the people there to keep track of sales opportunities. I expected a big positive reaction on Charles' talk - a large oil company interested and really spending money to forward Open Source ...

The crowd just seemed not very interested. My guess is because they were 90% academics. Is Open Source not interesting for the Oil&Gas industry? Hopefully, this is just a big mistake and the closed-source people simply don't see what's coming their way. I sincerely hope so.

Dependencies

I have tried hard to get KDevelop going on CentOS 6. And I wasn't the only one - I saw that it also happened to some people who do such things often. Yes, I have some experience with it myself, too - so I tried it anyway. But this time, I couldn't get Cmake to find Qt. I gave up.

When it doesn't work, the cause of failure is usually a problem with the dependencies. Software engineers know this. Pure developers tend to use as many external tools as possible, but software engineers try to minimize the dependencies. The more experienced you get, the less you like dependencies. As long as you get the services of a perfect system that guards all dependencies you're OK. But it breaks down. What kills is not only that you cannot find the right versions, but also (slight) changes in interfaces or implementations.

Moderns systems are riddled with dependencies. On the OS for one - something you cannot avoid. Luckily OS's are managed by others, and they often do that quite well. As an app developer you will have to counter the annoyances of OS changes, but these are usually not so bad. Even better, OS's provide more an more tools that used to be in separate libraries that you had to manage yourself - separate dependencies.

The decision to start using an external tool is a very important decision. Beginning developers often don't care at all and try to find something for every problem they encounter. They don't bother that they're using just a tiny piece of something, and have to take in a huge mount of dependencies. This quickly results in a dependency nightmare. That's why at OpendTect we actively kick out things as soon as we see that the usage less than strongly required. For example, we did that with FFTW (the MIT FFT library). The benefit of a few percent extra performance in the FFT routines did not outweigh the hassle of the dependency. While kicking out you often diagnose your whole situation, and that makes the operation even more beneficial. We based our FFT on the SU way of doing things, and then found some other things we could do smarter. So in the end we lost the annoying dependency, and FFT-ing is faster than ever before.

For Agile, you don't have to use loads of external stuff. You do have to have basic things like GUI building, 3D visualization, zip/crypt, and so forth. It's important to figure out what can satisfy your needs, but remember that one of your needs is minimal dependencies! Very often your bare needs may be satisfied best using a mix of many small packages. But in the end, you can use much less 'luxury' tools and take in packages that have other good properties like light-weight (no fluff), integrated (just one, rather than many dependencies), robust (maybe not so slick), stable user base (will be there tomorrow, and will still support a changing outside world).

And then ... after a few years you discover that you still bet on the wrong horse. But that's life for you. Murphy's law.

Meta-procedures

Procedures may be useful, but what procedures do we need? This is where what I call meta-procedures are for. How do you define a new procedure, how do you change one, how can you drop one? This must be some sort of community process.

In many organizations the meta-procedure would be: procedures are changed by your boss(es). Needless to say that this is not very motivating. That's why we have something like this:

• if you're not OK with a procedure: speak up.
• if there is consensus that you are right, then we can start a discussion on how to change the procedure
• normally, and while the procedure is not changed: comply (even if you disagree)

You can elaborate on each point (like: what is consensus? who will have the last word, etc), but the idea is clear: we use an established procedure until we decide, as a group, to change it. Also, if you are uncomfortable with a procedure, then discuss it. Chances are that either everyone has the same problem, or that others can enlighten you on the true reason for the procedure. In no case just spoil your and others' mood without any chance for them to plea their cause.

Can meta-procedures be changed? Sure. I can imagine that as your team grows you may want to break with the tradition because upscaling simply requires other decision making processes. Going from closed-source to Open Source may induce something like that, too.

Procedures and creativity

I have been working like mad in the past weeks. Couldn't get myself to writing stuff for this blog. But there's a curious thing I found out in the past years that I want to share. It's the paradox that creativity needs good procedures. Creativity thrives in structured, safe environments.

I read about this topic about a year ago, when the outcome of a study on this issue was covered by all newspapers at the time (here in the Netherlands that is). The surprise was that successful creative companies work well-ordered. The surprise was with the journalists and maybe most of the readers, but for me - no surprise at all.

In our software development team, we are constantly looking for better procedures to handle the standard tasks. We constantly search for better ordering of our working environment. This is engrained in our work structure, and we fight chaos everywhere. Still I might say that we make very creative products, we have literally hundreds of ideas still waiting to be implemented.

With the standard things handled in a standard way you can put your attention to where it pays off: in the areas that matter. No wasting of time and brain cycles on stuff that shouldn't take more than the absolute minimum of your time. If we all adhere, then we're a well-oiled machine producing with optimal performance. Another way to look at it is that good procedures give a feeling of security in the place you live in. We live in our software for a large part of the day. You want to feel safe in your home, you want to know where to find what, and what to do in what circumstances.

The weird thing is that the outside world often thinks of us as being the chaotic hackers. OK, we may not be as orderly as accountants, but in terms of structuring our work we are not bad at all. Oh maybe you are. In that case - change! It pays off.

Becoming a writer

You're reading this. That means my language must be - at least - bearable. I'm well aware that it will never be great. This is partly because English is not my native language. Now that will never change. Also, I lack talent. That won't change, either. What can change, and what has changed in the past years, is that I've become much better at producing reasonable stuff. And I'm faster, too.

When I press my team members to start getting good at it, they often say they are not so interested. They want to program, design, test - and so forth. I think that's a big mistake. If you want to become a great software maker, then your communication skills need to be great, too. That involves many many skills, and the ability to create good texts is one of them. It's not the least of them either. And (this must sound as music to Agile ears) it's re-usable! You'll find these skills useful in many parts of your work and life in general.

It is so important that I always press people to get on with it and do it. The complaint that you're not good at it won't hold: you were not good at writing programs either, years ago. How did you manage to become the programmer/designer/[your job here] you are today? Indeed, the usual way. Studying the art, and doing it - a lot.

So don't try to hide when a text is required from someone but take the opportunity to invest in yourself. "You're worth it".

More worries

I've talked about the necessities of someone worrying about the project before. What should be worried about? Well, that the project is a success ...? But what does that mean? The right things have to be made, and in time, with a certain amount of resources.

Finding out what those 'right' things are, that is a big thing - one of the main raison d'être's for Agile methods. What must be made, and what should it look like so to say.

One technique that helps the developer a lot is to imagine you are now the user. Give yourself a certain task and try to fulfill it. In a way, imagine you have a project that needs this new piece of software (that stuff you are now making). Gather the data and start producing results. Find out how hard it is to use your new stuff. That some things can't be done. Some parts are useless, other parts non-existing.

I call this (and I'm sure I'm not the first) putting on the user's cap. That helps you create a decent 'version 0' product. One that will not be pushed aside with disgust (at least, not immediately).

In some cases that's not it. There may be different types of users. Each with their own cap. Your stuff may be used by end-users and system administrators. Geologists and geophysicists. Managers and technical people. And so forth. Some projects call for imagining the needs of many different types of users. That's a lot of caps, and constant switching is often required.

To support all these roles you may be able to add options, or some extra tools. Sometimes, the roles are so different that you have to make separate programs or branches. Of course that will usually not mean you make different internal objects, just that you offer the services in varying ways. This is a big difference with waterfall analysis that already splits up the functionality from the start. That makes it almost impossible to later join the branches. In Object/Service-oriented environments the counterpart of this 'early split' is lurking. It's the tendency to offer everything as you implement it, rather than making tailor-made things for each of your user types.

Worries

Any project needs at least one person worrying about it. Some call these people 'project managers'. These poor souls do a lot of things that have very little to do with actually creating software.

Every developer except the total freaks will sooner or later get more and more project management tasks. Some people like this, others don't. Alternative career planning involves the 'internal consultant' route. It's also possible to become an 'external' consultant, but that could bring you into even more trouble with all sorts of management tasks. Very few manage to keep out of project management altogether.

One of the painful parts of project management is that, at the end of the day, you feel you haven't achieved a thing. This is not at all true, but it sure feels like it. It's much more satisfying to check in a nicely running piece of code or a nifty framework design. The thing is that project management can be seen as charity, on a project. You are enabling the others to just have all the fun. While you're sweating on e-mails, reports, checking progress and so forth, the others can enjoy themselves and make great stuff. While you are worrying, they create.

It's a rotten job ... but someone's got to do it ...

Design vs the real world

Some more on the averages thing. For Agile Geosciences (let's call that the real world), averages are a concept by itself. All sorts of averages are related by the common denominator: calculate a central tendency.

If you look in OpendTect, you'll see this implemented in a Stats::RunCalc object. In design, it is much more important what type of operation you are doing, than what it's used for. So, we have an object that can calculate running average but also min/max, standard deviation and so forth. The interesting part here is that in that object, the median is the outlier. All other operations are done in a similar way, only median requires buffering numbers and sorting.

This is a common 'problem'. The solution model (the design) has another conceptual structure than the real world. This fact crosses the idea behind all the 'unified' approaches I have seen. These invariably see the design as a sort of transformation of the analysis. The 'problem' explains why there are almost no successful code-generation-from-analysis-models around.

If you look at old applications, you can see that progress in software development is ongoing, and huge. Quickly made amateur programs can do so much more than multi-man-year programs from 20 years ago. This is in no way related to advancements in the code generation world. It's the result of a bottom-up process of making the developer more powerful in his design/implementation process.

Advancement in software development comes from the implementation side. Successful applications spawn more generalized toolkits. These in turn enable the building of even higher level applications. This is how we can now make things with 3D visualization and modern GUIs in the time you could, years ago, make bundles of hard to use, limited functionality batch programs.

Flexibility vs ease-of-use

Ah ... the sun was nice - a memory only though, being back in the dark&windy Netherlands. Gran Canaria was also rather windy, true, but at 23 degrees C and nicely sunny that's a lot easier to handle. And ehhh, what beautiful islands they are, the Canary islands. No geological stuff on volcanics on this blog, though.

I only read, via the mail feed, some of the latest Agile Geosciences blog postings. Some were about things we didn't have in OpendTect. The cepstrum, integrated traces, all sorts of averages.

That is: we had all the tools in the code, or already available via the UI - but not easily, readily available. This is a nice illustration of a common issue in software development: flexible vs easy-to-use. Very often, flexibility and ease of use are antagonists. Very flexible tools are often very difficult to use. And the other way round.

The Agile way - if there is one - is to try to get the 80/20 benefit in both domains. 80% of the benefit for 20% of the effort. That. magically, will often result in an optimal usability vs flexibility.

In any case, in the next OpendTect releases we will go for ease-of-use for the cepstrum (i.e. we put in programming effort to make it single-click), integrated traces were already so easy to do that a mailing list post on the 'how' will do, and the averages are easy for the ones that are important in OpendTect (average, median, RMS and most-frequent).