Perspective, Software Development

For the love of the User

Software is for the user. It is not for the Software Engineers who develop it. In the end, software will succeed or fail to meet user needs. The user is the arbiter of software’s fate. Oddly though, many software developers tend to resent their users. The users are prone to strange behaviors. Sometimes they can even come across as whinny children to jaded developers. But we must do away with this flawed way of thinking. We must act as humble stewards, gentle of heart, and eager to please.

Users are the life blood of a software product. Without them, the product will fail. As a result their needs are paramount, and must be address to the best of our abilities. If this is the case, then why are developers so often frustrated by their users? Remember we are fluent in the machine tongue. Generally speaking, users aren’t. Sure they can use the machines, to a limited degree. But they don’t understand them like we do.

Imagine you are in a foreign country. The only way to get your work done is to cajole a lumbering beast into action for you. Without understanding the beast’s language, even simple tasks could be infuriating. Users who are less familiar with software might feel the same. Only remember that we specialize software to particular tasks. As a result users need to learn, remember and use a variety of these ‘beasts’ to get their work done. Also remember, they are being evaluated by their ability to get work done, using your software.

And so scared, frustrated, and feeling impotent, they turn to us. They wonder why their actions did not work. They ask for strange features or work-flows. All these feeling arise because they don’t understand their tools. Sure we could ‘educate them’. But if the way to use a tool is less than obvious, or they only use it seldom, then you can expect them to forget. Not to mention, you have to convince them to take the time to get trained, rather than working. Even we don’t feel comfortable trading training time for working time. So why should we ask that of them?

Two paths remain to us. We can tell the user’s they are wrong and constantly bicker with them, trying to explain the proper way. Or we can choose to listen. The way we thought was obvious is not. They need more help, because the grammar of machines is difficult. I would call this path ‘Stewardship’. We have to think of the code as belonging to the users, not to us. In so doing, it becomes clear what choices we need to make. If the code is for the user, then their needs overrule ours. If they aren’t fluent, we must may the software more approachable.

We are like gardeners. The land we tend is not our own, but still we make it bloom with brilliant flowers. We cherish the blossoms, and suffer when they are trodden upon. But the garden is not for us. Imagine if the gardener chased off the owner with a spade when he ask for a new row of lilies. The gardener would be marched off and a new one brought in to replace him. This is not an exact analogy, since users pick their software. They might just avoid a certain gardener altogether.

If instead, we are gentle and approachable, we could better tend our gardens. If no one ever walks our garden paths, then we put to waste all the love and beauty to garden contains. Software without users, despite its brilliant design, and delicious complexity, is dead. If we want vibrant, living software we must serve our users. We cannot lord our understanding over them, but must instead steward the code for them. With gentle hearts, we can learn their needs, and make the garden they need. In the process we may discover an even greater beauty.

Software Development

If you give a Dev a board game…

From my first lecture on C, I have been tinkering with side projects. I’ve done projects purely for exploration and entertainment, like a text-based adventure games. More recently I’ve done utility projects like a script to correct QIF formatted text. Recently I took on a project of a larger scope.
A while back,I read an article about a simulation of Machikoro. It is a ‘city-building game’, with rules that are easy to translate to code. In particular, the idea of using the simulator to ‘evolve’ an optimal strategy for the game captivated me. This was applying Machine-learning to a board game. I figured ‘I could do that’, and got to work. I encountered many distractions and set-backs, including a new baby. But this month I am pleased to admit that I have hit a milestone.
To support the ‘evolution’ aspect, I had to be able to run thousands of simulations in a reasonable amount of time. And after a bit over a month of concerted effort, I made it. I took my code from being a collection of classes to a library and simulator able to run 1000 games in 15 seconds.
I started back in December with classes to represent the deck of cards, a strategy for play, and a player state. The first step after this was to create a basic AI* to act upon the player state, and a given strategy. Borrowing from the article I had found, I decided to make the strategy more static. The decision logic reduced to constant decisions like ‘always yes’, or ‘always the cheapest available’. Then the AI only needed to use the Strategy to answer queries from the Game.
*Note: I am capitalizing and italicizing Class names for ease of identification.
After the simplified AI was complete, I got to work on the Game, which would simulate a single game. I decided that I wanted to use fluent APIs to instantiate a Game. I spend a good chunk of time to get these write, but it helped to make the main routine clearer. While I developed the Game, I decided to abstract the mechanisms of the game. This allowed me to separate the calculations from the sequence in which they are applied. I extracted the Engine to handle things like calculating which AI if any has won, or how much money this AI gets with this dice roll. Meanwhile the Game can manage whose turn it is, and who rolls the dice.
Testing both the Game and the Engine were somewhat arduous, but it was time well spent. I caught numerous bugs, and infinite loops before I ever ran a full simulation. Thankfully the Deck, State, and AI were all similarly tested. But I do wish that I had adhered more tightly to TDD. Instead I was very eager to getting the core functionality working.
Once these pieces were in place, I initiated my GitFlow, branching Master, Dev, and a new Feature. After pushing version 1.0 to Git, I started work on a new Feature, multi-game simulation! And while I tinkered with a Simulator, I realized that my fluent APIs had a bug. So I went back to Dev, and produced a Hotfix, which was merged into Master. From there I re-based the Feature, and continued my work.
With the Simulator, I needed to initialize a Game, but also to be able to run it N times, without interference from the previous rounds. So I had a two-pronged approach, I would accumulate the results of each game, and I would allow a Game to be reset. Learning from my forebears, I was sure to include randomization of the first-player when I reset. This removed the skewing of First-move advantage from my results. With the core Game working and fluently initialized, I was able to simple inject it into a Simulator to run.
The original simulator was able to run 1000 games in around 80 seconds. This performance is alright, but my personal dev box has 8 cores and the Simulator was maxing out just one. So to improve performance , I began to look into Python multi-threading. I found two similar flavors of concurrent operations in Python.
I elected to try Tasks first, as it seemed similar to Microsoft’s Task Parallel Library. Sadly I was not quite right about that. The BatchSimulator’s performance was terrible. For some reason it never used multiple cores. The original time for the BatchSimulator was 150 seconds for 1000 games. While it is likely this was user error, it was enough to discourage me from pursuing Tasks further.
So I turned to concurrents. And with concurrents, I had much better luck. In this case I spawned some sub-processes. I created the Coordinator to provide each fork with its own copy of the given Game, and an assigned number of games to run. Then each fork created its own Simulator, and ran the given number of games. Once each Simulator completed, the Coordinator would accumulate the results. After all the forks completed, the coordinator calculates the final statistics. This provides an overall winner. To make this easier, I extracted the SimulationResults class. I then added public methods for merging and calculations. By leveraging sub-processes, and existing code, the Coordinator was able to run at least 1000 games in ~16 seconds. Now I say at least, because the Coordinator divides the games evenly among the sub-processes. So to ensure that at least 1000 games are run, it must round up on the division of games per sub-process. But having more data is never a bad thing.
I was able to push and close this Feature recently, and I am very pleased with the progress. I went from single game simulation to rather performant 1000 game simulation in a month. I now have something to show for my ideas and my work. This milestone leaves me at a good break point. I can either continue working on the simulator to pursue the machine-learning angle. Or I can change focus and return to this project later. At the moment, I don’t know what direction I will turn. But I wanted to take a step back and look at what I have accomplished, and share my ‘geeking out’ a bit.
If anyone is interested in the source, you can find it here.
Journeyman's Digest

Journeyman’s Digest 01

Issue 01

This Issue’s Highlight

The Steak-dinner clause – Imagine you are in a tense business negotiation. You’ve been at this all-day and no one is willing to budge on who has to pay a fee. As you close for the day, you realize that if you don’t resolve this issue tomorrow, there won’t be any agreement. This was the author’s situation when he happened upon a brilliant way to handle the disagreement. Moreover, their resolution not only helps get the contract through, it also brings the two companies together over time, creating a very strong business relationship!

The Goods

The Tales of a White-Hat – A White-hat finds a massive security hole in the popular image sharing service, Imgur. This is his story about how Imgur’s bug bounty program worked, and his journey with them to make the service and the compensation for finding such bugs better. Bonus Link: HackerOne , the Bug bounty board mentioned by the author.

Log Structured Merge Trees – This article describes the basic working of a Log Structured Merge Tree, which is a happy middle ground between Journal/Log and Fixed Index ( like Hash Table) data storage. It is intended to provide good write and read speed, by balancing the concerns and attempting to improve the linearity of the Read operation.

Knowledge-sharing Architect – What is the Architect’s job? Should they be responsible for implementing the core framework, only to be pulled into another project in the starting stages? Or should they be mentors, mental-giants who slowly educate their team on the intricacies of the current application, so that nothing in the big-picture gets ignored? This article discusses the common arguments regarding the Architect’s role, and proposes an alternative which I believe solved the existing problems in a more feasible and sustainable way.

Ubuntu has gone mobile – Ubuntu has apparently released a new phone! It’s price is comparable to the average smartphone, but it comes with some interesting features. This review goes over the various software changes, and capabilities. Personally, if and when the UbuntuPhone comes to the US I may consider purchasing one.

Scythe – Most-Hyped board game of 2016? – ArsTechnica loves it’s boardgames! And they have been drooling over a new game called Scythe for most of the year. It was recently release, and their review is glowing. I think the game sounds excellent and totally worth the rather high-price tag. Check it out and see what you think!


This Issue’s Curiosity

Free Reverse Engineering Textbook -The kind Mr. Yurichev has compiled his notes and his method for reverse engineering software. I was only able to read a small fraction of his material, but a college course based on his work would be thoroughly interesting! Please feel free to poke around his free textbook!

Journeyman's Digest

Journeyman’s Digest

Issue 00

This Issue’s Highlight

Hacker News – Hacker News is a service similar to Reddit. As I understand it, there is a community which submits links to various recent news, and then by a voting system the best finds rise to the top 30 entries on the list. Through this community, I have found many great articles, including many which I am including here. I strongly encourage you to check it out!

The Goods

Some advise for new managers – This is a short management snippet for new managers. It regards how to advise and discuss the contribution of your employees. I could summarize it in one line, but I think the effect is best if read in full.

Ranks of popular programming languages as of June – In June, the kind posters over at RedMonk took the time to analyze the contributes and discussions in GitHub and on Stack Overflow to determine how various popular languages are doing in terms of use. The big five this time include Javascript, Python and C#. Be sure to check the list out for more details on their methods and to find where your favorite language ranks!

A JSON coding horror story – A young developer gets hired on to a new company. His experience starts as any new job does, learning the new systems and trying to understand the code. But as our hero learns more, a growing horror begins to loom. Will he escape it’s clutches? Or will the code claim another victim? Look here to find out!

A Discussion of TDD – This is a forum thread discussion of TDD, started by a frustrated programmer trying to convince his company to adopt a better practice. The original poster asks for specific reasons and explanations for why his company should do TDD. There are the normal responses about making the software more stable, and more robust. One response in particular stood out to me, as it described instead how to explain these values to a Manager, in terms of time saved and money saved/spent. It is overall an insightful discussion for anyone in a similar situation.

The dark-side of UI patterns – This is an hour-or-so talk about the dark UI patterns that can be seen all over the web. They do make money, but they are predatory, generally immoral, and in some cases possibly illegal. The speaker describes many of the type of patterns which are employed. But he goes a step further and offers an explanation for why the patterns arise in the first place. As always it comes down to money. He then concludes with ways that the industry could help to dissuade and possible eliminate the use of these dark patterns.

This Issue’s Curiosity

Uber migrates to MySQL – This weeks curiosity regards Uber’s move from Postgres to MySQL. I wasn’t able to dedicate as much time to looking into this article as I would have liked, but the first third or so was full of interesting technical discussion. I hope you will enjoy it too!

Software Development, Work Projects

Licensing Overhaul – Return of the Designer


In case you missed parts one or two of the series you can find there here: Part 1- A New Challenge, Part 2 – The Whiteboard Strikes Back

Welcome back to the series on Licensing Overhaul! Last time we witness the rise of a new system, and discussed the gritty details of some of the more important components. This time we will review our journey, and focus more on some realizations brought on by hindsight.

To summarize, my PM wanted to change what some of the existing Licenses allowed our users to do. So I spend some 6 weeks digging through the old jungle of code, and developed a simpler, and in my opinion cleaner solution.  Last week, I went into detail about this design. This week I plan to cover the benefits that were realized in the design as well as some of the foreseen drawbacks. I will address the drawbacks first.

Every decision made in a design naturally chooses a set of advantages and disadvantages to take for the software. My new licensing system is no different in this respect. While the design supports a great number of data types to query, it is somewhat weak in this area.

Specifically the performance of the query may be adversely affected should the number of datatypes be greatly increased. Thankfully, the number necessary would be very great indeed. Each License would need to support something on the order of 10,000 different data types before any notable performance hit would be noticed.

This weakness results from the License Checking each ObjectType for its data type. This could be easily fixed if the License were to use a data type to Permissions dictionary rather than a simple list. However time has not yet been allocated to for this change, and the solution was not discovered until after the project was determined to be complete.

In a similar way, the number of Licenses may also present certain challenges. Since each License is queried for each permission request, if the number of different licenses checked out by the system should increase greatly, it would also negatively impact the performance of the query.

Again, the number would need to be very great, on the order of 10,000. However, the number of License checks has a multiplicative effect on the ObjectType check. As a result, it would probably be best to spend the time to fix the ObjectType rather than the License check bottleneck, as this would offer greater improvement between the two.

So much for the weaknesses of the design. Now on a happier note, the design has several strong benefits, some were realized very early on in the implementation, while others became apparent towards the end. I will start with the one which became obvious while I was implementing this change.

Perhaps the most long lasting of the benefits for the new system is that it was built to support incremental changes! The new system did not completely replace the existing License checks, rather it came in along side them and took over a select few. By doing so, the scope of responsibility remained manageable during implementation, and it only touched what it had to. This benefit was by far the biggest in terms of controlling scope creep.

The next benefit became obvious as I drew to the end of the project. By that time, the accumulation of permissions knowledge grew to such a size that it was easy to see the patterns of allowance by the licenses, at least for the affected area of data. Since we now had an aggregation of the License permissions knowledge, future changes to said permissions are much simpler. Rather than having to dig through many thousands of lines of code to replace the various license checks, one would need only change how the License Object was created in order to implement the desired change.

To cap things off, I realized that there was one further benefit that could be realized. However this will likely not be until the remote future. At present the Mediator relies on the Manager to determine what Licenses are checked out. Presuming that any replacement manager implements the same interface, it can easily be swapped out when the time comes. And with the code base gradually moving towards C#, it seems that it may be inevitable, albeit sometime in the next 20 or more years.


All in all, this was a very enlightening project to be allowed to work on, and very rewarding for the chance to implement a system which may one day widely affect the application in the future. Naturally there are some optimizations that I could chosen had I been thinking about performance, but it has been said that premature optimization is the root of all evil.

As always, I thank you for listening and I welcome your thoughts and comments! Let me know if you think of any ways to improve the design, I would really enjoy such a discussion! And until next time, good luck and God bless!

  • Bottleneck road sign –
  • Library Interior –
Software Development, Work Projects

Licensing Overhaul – The Whiteboard Strikes Back



In case you missed it, you can see Part 1 of the series here: Part 1 – A New Challenge

Last week, I introduced my second biggest work project to date. Shortly after re-vamping the License structure, my PM decided he also wanted to change what some of the licenses allowed the user to do. Upon digging into the code, I discovered a tangled web of multiple checks and confusion. I thought there was no way I could finish in 10 weeks. Thankfully my estimates were in correct, and some solid design work saved the day! Today, I will discuss the design aspects of my new implementation.

Starting out, I knew this project would be not only big and rather difficult but that it would also be a catastrophe later if I did not do it well. Even so I was surprised by just how ornery the project got before the end.

To be certain, I knew at the outset that I wanted my design to be better and if possible to be more Object-Oriented that the previous implementations had been. However I had very little idea what that meant at the time.

As I progressed through the project some goals did eventually become apparent. I am listing them here, in the hopes that I might learn to generate these design goals earlier in the project going forward.

  • I wanted my design to provide an explanatory interface to the user. I wanted it to be clear from looking at a simple call what permission was desired.
  •  I wanted permissions presented to be general enough that only a few would be needed, and that these would be clear in their intent.
  • I wanted to ensure that any code written in this phase was stable enough that it wouldn’t change should any new scope be added. To be more specific, if the covered permissions or the data types that were covered ever increased I wanted the present code to remain unchanged, both in syntax and in outcome.
  • I wanted my code to be segregated enough that if ever the original License Manager need be replaced, it would be done with a minimal effort to update the new code base, and without disturbing the Permission request calls listed above.
  • Lastly, I wanted my new code to be easy for anyone who came after to learn and to use, so as to reduce the multiple versions of the same License check that were seen in previous implementations.

Again, I admit that these goals were not all so verbose when I started the project, but I did set out with something very like this in mind as I began to design my solution.

During the first few weeks of the project, I spend a great deal of time sifting through the code, concentrating on the Main UI area, and on the areas which my PM had mentioned he wanted to change. I made note of the patterns, and anti-patterns that I found throughout, and used these to inform my design choices.

While sorting through all of that information, I also spend a lot of time at a whiteboard drafting, and redrafting the objects and responsibilities that I wanted to manage, in order to solve the problem.Overall a great deal of attention as spend on their interfaces. The picture below shows the resulting system that I developed, as drawn on a whiteboard.


The workhorse of my solution was the Mediator, which was responsible for routing the Permission Request from any end point through the licenses that the system currently had checked out, and provide a Boolean response to either allow the request or to deny it.

To facilitate its work, it is injected with the License Manager, which is used to determine what Licenses, if any, are presently checked out for the system, and then it called on the Factory. The Factory, per its namesake, would create License Objects prepared to answer the Request queries being routed to them by the Mediator.

All of this architecture was created to support the abstraction of the idea of a Permission, that is an allowed action on a particular data type. A License in this sense, is composed by the series of Data types and their allowed actions under that License.  The Connection between Data Type and Action Type are represented by the Object Type object.

Now, some of the more seasoned developers reading this will probably be shaking their head at this name, but I assure you there was a good reason for it. In the first place, the code base I had to work with had another meaning for the term ‘data type’ and as a result the only other suitable term that myself, and an English major could conjure up that meant something similar to our intention was ‘object’.

Getting back on topic, there are several classes where a 1 to N relationship is specified. In laymen’s terms, this means that it is possible for multiple of the ‘N’ type objects to housed under the ‘1’ type object. This is a oversimplification, it will suffice for the moment.

The reason there are several 1 to N relationships is because I wanted to abstract the responsibility for know whether or not a particular action was allowed, while simultaneously providing a simple and easy to use interface for such a query. What I ended up creating is best show by the “call-stack-like” write up shown in the picture below:


As I mentioned earlier, the Mediator would route the query to the Licenses which were checked out at the time. So each license is asked whether or not it supports the requested action. To determine this, each License will in turn find the matching Object type of the query, if it has one, and will further route the request. The ObjectType then response based on whether or not it has permission for that action.

Since the number of licenses is unknown, some additional check are needed. For example, it makes sense to just skip the checks if no licenses are checked out, since there would be no response. However if multiple licenses are present, then some rationalizing needs to be done on their multiple responses. Thankfully, the PM decided that it was sufficient to allow the action if at least one of the Licenses that was checked out allowed it.

That is really all the more complex it got. After the appropriate associations were made to the various license. This basically concludes the design portions, and data types, it was a simple matter of tracking down what code I needed to change and how.

Admittedly, that process took nearly a month, and would have been a disaster had it not been for some stellar QA help. But overall, I was able to finish the vast majority of the code in about 6 weeks, and the QAs were able to catch up and feel secure in their approval around 8 weeks. Naturally being nearly a month ahead of schedule made the PM very happy!

So this week we covered the goals of my new design, the general responsibility of the software objects that I used to create the new system, as well as the Request routing hierarchy. Come back next week, when I’ll be discussing the concrete benefits of the new design, as well as some of the draw backs. Thanks as always for your time!

Part 3 – Return of the Designer




Software Development, Work Projects

Licensing Overhaul – A New Challenge




Anyone who knows commercial software knows the dreaded self-mutilation that is software licensing. From the developer point of view, having the application turn off parts of itself always seemed a little silly to me. But from the business standpoint, it is not only necessary but it is of the utmost importance to generate revenue and to protect your product.

Additionally, Licensing is one of those things that touches everything in the application at least at some level. As a result it can be terrible if you get it wrong, and worse when you have to make changes to it.

For my second stint on a legacy code team, this was the exact task they assigned to me. Honestly, when we started I didn’t think there was any way I could get it done in the 10 weeks I had to do it in. To help get you some idea of what I was dealing with, I will start by sharing what I found out about our previous system.

Now to start off, you should know, this code base is in C++. Furthermore, it bears the scares of a transition from C into C++ and from functional programming to Object Oriented Programming. In laymen’s terms that means that this code base is older than I am. A lot older.

To be certain this code has seen some years, but the fact that it has stuck around this long is a testament to its impressive library of functionality. After all, no one keeps code that doesn’t do something useful.

Getting back on point though, the previous manner of managing Licenses reflects the code’s age as well, though perhaps less favorably. In previous generations of this product License were sold per feature area, rather than as a bundle. The result was that many areas of the applications, and especially the central UI area, are thoroughly riddled with checks for multiple of these Feature Licenses.

Admittedly, the LicenseManager object does a good job of executing its apparent job description. The trouble is that since the a particular function might be available in multiple Feature Licenses, it can be difficult to tell, from the back-side of the UI, what the program is trying to protect from the user.

For the work I did on this system, the proliferation of multiple License checks was especially troubling. You see, recently my company had reorganized the Licenses which they sell, resulting in some confusion between the old and new Licenses in the code. For simplicity I will call them License A, License B, License C ( which is really a combination of A and B) and License D.

My product manager wanted to allow the users under License B to see data which had been created in the same project by users under License A. Further he wanted them not to be able to modify this data, or to be able to import or export it. Given the current system and confusion, this was a very tall order!

As I have mentioned above, the previous system required the UI area to make a number of checks for several different Feature Licenses before determining whether a particular function was allowed to the user or not. Naturally, this resulted in several complex logical expressions in an attempt to determine whether or not the user was allowed to take a particular action.

archaeology-clipart-confusionFor example, in order to determine if a user is allowed to load a given data type from the database, the data manager would first query the License Manager for Feature Licenses X, Y, and Z. Then if the user had X or Y, but not Z they would be allowed to proceed to loading. If they have Feature X they loaded a particular subset of the data, and if they have Feature Y they loaded a different one.

Admittedly this is a simple case, but imagine if this kind of check was made in several different areas of the code. The same check written by different developers at different times for slightly different reasons. The natural differences between coding styles would prevent one from simply searching through the code base for all similar checks.

Furthermore, there is no clear place in the code to determine what a particular Feature License would allow or disallow. This is even further compounded when the new Licenses do not directly match up with a set of Features, and under the hood  they are really just re-skinning  some of the bigger Feature Licenses.

But that is enough of my belly-aching! After sifting through the old implementation, and some whiteboard work, I was able to develop a better design. Please check back next week, when I’ll explain how I made it easier to make the license checks, easier to tell what a license allows and doesn’t, and how I was able to avoid replacing the entire licensing system!

Part 2 – The Whiteboard Strikes Back!

Part 3 – Return of the Designer