This is not my normal fare. If you’re not a computer geek you may find the following paragraphs a little bit technical and quite possibly uninteresting because of that. I’d encourage you to read on though as what you should come away with is a new way to look at the problems you face and a strategy for dealing with them that will bring you much personal satisfaction or at least will cause you to pull the least amount of hair out of your head as possible.
There is never anything really new in the world of computing. All we have are problems that have been solved before and new flavors of those same problems and solutions. What really changes is that people forget that we’ve already solved all of the really difficult problems many years ago. We had to because they were new problems when computing was something fresh in industry. Now that computing is pervasive what we have is a repeating cycle of identifying problems to be solved and figuring out how they’ve been solved before or ignoring the past (at our peril) and creating entirely new solutions which are in fact, just different colors of the same solutions we came up with before… if we’re lucky. That amounts to a statement like, “Well, we have a really complex problem, so here’s a stunningly complicated solution.”
I, for one, detest the idea that complex problems need newly invented ultra complex solutions simply because the problem appeared superficially (or actually is) complex or new. There is no problem so complicated that a very simple solution cannot be identified if you think about the problem the right way. There are insanely few problems which are in reality the least bit new. At best, they’re just the same problem in a new shape or color, so to speak. In a moment, you’ll be introduced to my preferred method of solving problems which always yields fairly simple solutions. It does that because it works like the thought process of early Macintosh computers. Early Mac’s were built; seemingly, with a notion something like, “Give them so little memory and processing power that they won’t be able to do anything anyway.” I must at this point give a wink and a nod to Douglas Adams who originally made that exact statement and from whom I’ve borrowed it. There’s a certain amount of sarcasm in that but hang with me and you’ll see my point.
What I mean by all of that is, simplifying the problem comes down to really seeing where the actual fundamental problem is (Mac users, of which I am one, wanting to do very intensive computational tasks on end-user grade hardware is the fundamental problem.) and not where the superficial problem is. In this case the superficial problem is one of Mac’s being the preferred platform for those doing computationally intensive tasks; like video editing for example because they’re user friendly, as opposed to Windows which is user unfriendly and UNIX/Linux which is downright user-hostile. UNIX/Linux server-grade hardware would be the right way to do these computationally intensive tasks but they suck to use for humans. So Mac users are the fundamental problem. They picked the wrong tool. Apple responded by making sure that the user would realize that and would eventually put those workloads onto higher end hardware. Now we have video editors doing very small bits of editing on very small bits of video on their Mac and then sending many such snippets to a larger compute cluster for rendering and final processing to come out with a whole “thing”.
Those familiar with “Grid Computing”, “High Performance Compute” and other flavors of the topic know that what you’re really dealing with is a system that understands bounded resource blocks and workload. What it amounts to is you have a bucket of resource (CPU/Memory/Disk/Network) capacity and a bucket of workloads that have a discreet moment of being started and which will run to “completion”. You want to dispatch computation jobs to be executed, allow them to run to completion and then report on the status and resources taken to accomplish that. What you don’t want to do is worry about uneven load profiles, manually intervening when jobs fail or systems lean over, or figuring out which host to execute a job on.
Some systems like LSF/OpenLava and others were created back in a day where there was a huge variety of capability as far as horsepower and there were lots of proprietary hardware platforms. Those factors joined with factors like making sure that software licenses which were few in number were always in use, fair share allocation of computational horsepower & software licenses and organizationally induced prioritization of this project versus that project.
Today, hardware performance is orders of magnitude better and we’re not so much worried about computational horsepower so much as footprint cost efficiency. Back in the old days we’d run on-premise clusters of large numbers of very expensive servers in very expensive data centers. Nowadays we Cloud Service Providers which can provide enormous amounts of extra computational capacity on-demand which can be spun up only for as long as it’s needed and then spun down immediately afterward to minimize run costs. We’ve eliminated the sunken portion of data center run cost from the equation.
As we all know, most of the really great inventions in history were made by eliminating something from a prior invention: A magnificent martini is made that way by the elimination, or at least minimization, of the Martini (vermouth) from the equation. In the same way, eliminating the concept of owning actual servers and putting the load in the cloud enables organizations to radically alter the cost associated with operating high performance computation grids.
Kubernetes has the ability to dispatch arbitrary code execution to nodes. The cluster is aware of what nodes are part of the cluster and how much load they’re under so it’s relatively easy to code in a little Python/Ruby/C/Whatever to interface with a SQL or NoSQL database to build a list of jobs needing dispatch and to get them dispatched. When there becomes a queue of jobs due to lacking of free resources the code can, with very simple boundary configurations, elect to launch new execution node instances on the CSP (Cloud Service Provider) infrastructure of choice or to persist with the queue having some non-zero depth.
The efficiency to be gained is not simply in the fact that the company no longer has to own large numbers of servers and to pay for the continuous operation of those servers regardless of their being fully utilized or not. A huge gain is in the simple fact that CSP’s tend toward pricing based on utilization of network bandwidth and data ingress/egress from their assorted block or object storage systems but not from in-cloud usage of those very same storage sub-systems. The actual cost of the CSP provided CPU cycles, memory utilization and in-cloud storage access is heavily subsidized by out-of-cloud network/storage IO charges. High performance compute grids are almost universally highly intense in their utilization of CPU and memory and are notoriously weak in their need to import/export large amounts of data from the computational environment.
The next big change we see is that jobs are not actually arbitrary in large part. Many jobs are regularized. That is, they are routine and come about as a byproduct of the development process. When you complete a piece of code, it needs unit tested and regression tested. When you design an ASIC it generates follow-on load which is predictable. Many organizations rely on grid computing to run routine, regular reports, analytics and business processes. These are things that can be statically defined either in code or in databases. It’s a standard workload. Everything else is arbitrary workload.
So what we have here is an incipient change in how HPC gets done. The hard part had always been dispatching jobs. Now the hard part is architectural. Orchestrating job dispatch has been made trivially easy. Discerning what is a static job versus what is an arbitrary job and causing Kubernetes configuration to be automated is the current challenge. This is actually trivially easy to accomplish because of the ease of determining the static versus arbitrary nature of any particular job.
I’m not saying that there’s no effort in creating the necessary bits of code and building the necessary back end systems to accomplish these goals. What I’m saying is that we no longer need to pay IBM’s (or whomever) extortionist license fees for LSF (or whatever) and we no longer need to maintain extensive farms of servers, difficult to manage and highly specialized grid computing engines which require expensive-as-hell HPC experts like myself. All you need now is a basic bitch sysadmin who knows extremely common and popular technologies like NoSQL/SQL, Python/Perl/Ruby, Linux, Kubernetes, Docker, etc… There are maybe a few thousand people in the USA that really know how to make IBM’s LSF grid computing software work and to troubleshoot it. There are probably a million or so Linux sysadmins (also like myself) who know NoSQL/SQL, Python/Perl/Ruby, Linux, Kubernetes, Docker, etc… and even if they don’t know one of more of those things, they’re all easy to learn if you’re already a Linux sysadmin. They’re easy to learn for us because they were bloody well meant to be. If we’re to use them, and we’re a lazy bunch which is why we automate everything we can figure out how to, it has to be easy to learn, easy to use and easy to automate or we won’t do it.
So, now that I’ve given you this off book use case for Kubernetes, get out and use it. Yes it’ll take a few weeks longer than LSF would to implement but in the end it’ll cost you millions of dollars less to maintain and you won’t have to pay IBM’s (or anyone else’s) heart thumping-ly exorbitant license fees which are deliberately structured to extract every possible last cent from your organization.
Go (to heck Big) Blue!
I had to land for a four hour layover in Dubai during my recent flights to and from South Africa. On the way to SA as soon as we landed I went to the Duty Free which actually is the entire airport area near the boarding gates in Dubai Airport now. I needed cigarettes. Wasn’t going to buy them in California at 8 dollars a pack and I was told they were 2-3 dollars a pack in SA. I didn’t figure they’d have any of the rather rare smokes I like though so I checked around the Duty Free and found cartons of Djarum Black for $13.26US and immediately bought a carton and headed to the nearest smoking lounge.
After I’d finished my first smoke another guy, middle eastern looking, walked in and sat at the next seat over around the same ashtray as me. We exchanged greetings and queries of where you going, where you from and such. The were are you coming from question for both of us had the same answer, San Francisco Bay Area. Our destinations couldn’t be much different. He was going to Afghanistan and I to South Africa.
Turns out the guy worked with our boys in brown over there during the war’s heyday and had moved at the first opportunity to California where he loves the climate and culture and clean & safe everywhere sort of feeling. The last remnants of his family, I gather, are in Afghanistan taking care of the last of their property holdings.
We chatted for a long while about religion, politics, war and sports and found that despite our mutual distaste for each other’s system of beliefs that we were perfectly happy to be in each other’s company and to talk, entertain and educate each other. After a while both of us were hungry and we retired to the nearest place where one can get shawarma. Having never really had an authentic regional copy I asked if I should get the sandwich or Arabic version (a wrap lightly grilled). He suggested the Arabic version and so we ordered. The food place had nice seating but didn’t serve alcohol so we both had what they called tea though it was doped with sweetened condensed milk and cardamom so heavily that it really resembled the taste of horchata more than tea. I was unused to the flavors but tried to enjoy them without a preconception. The tea worked to soothe and really broke the shawarma flavor up. The shawarma was, according to my companion, terrible if edible. I thought it would be spicier but was otherwise getting the distinct sensation that a Greek had introduced them to the gyro but called it shawarma so they wouldn’t get offended at it not being a local delicacy.
In the end we ate till just before my boarding time and then I ran full out to my gate to make my flight. My companion, whose name is Ghani, turned out to live not 20 minutes from my house so we traded contact info and agreed that we’d try to find a decent shawarma and continue our conversation upon our respective returns to the States.
In almost every way, the pair is incompatible. Opposites in so many ways. The salve for that problem though was to simply leave aside distaste and to engage in open and honest conversation where we didn’t bog down in the rote definitions of words. We defined the words with contentious possible meanings ourselves and left the “well that’s not what I call that” over to one side. It really made the conversation entertaining and educational.
If countries, religions, political movements and most of all groups of people could accomplish what we did there would be no international conflicts and fewer conflicts within individual nations. Live and let live gets you only so far. Sometimes you really need to deliberately set aside distrust or dislike and simply coexist.
Hi Ghani! Be safe in your travels and let me know when you’re back in the States. This time the dinner is on me.
My recent trip to South Africa was partly to do a couple hunts and partly to see what the long range precision rifle scene there was all about and how it could be well served. What products and services were available, what are needed and what just won’t play there. They have their own competitive rifle sports that are very like some of ours but different enough to have special needs. They also have a budding long range precision scene in the more common American influenced sports like F-Class.
Along the way I met a huge number of people: Gunsmiths, Knifemakers, Suppressor Makers, Farmers, Competitive Shooters, Guides, Importers, Exporters, Collectors, Legendary Hunters & Soldiers and 3 of the toughest, most well behaved polite & intelligent teenagers it’s possible to raise in this world.
I took some time to teach a few basic ballistics classes for some friends and we even got down to doing some long range live fire work complete with challenging drills and tests of both bravado and ability. Shots under 500m were there but we focused on shots over 500m and I have to tell you, these guys in South Africa are not fooling around. They’re motivated, well educated and not afraid to spend money if it’s going to return results.
I learned not only what biltong is but how to make it and why it throws jerky straight into a cocked hat. I saw fully 15 species of antelope, along with warthogs, ostrich, cape buffalo, and more. I got to experience what it’s like to be an ethnic super-minority and a chance to learn a little Afrikaans.
Afrikaans is a lot like Dutch but to the ear it sounds like there’s a lot of Hebrew or Arabic language influence with some of the sounds. I found it greatly difficult to understand spoken Afrikaans with my hearing loss eliminating vast wadges of the audio inputs and lip reading it is right out, much like Hebrew and Arabic. I did manage to toss a bit of surprise around when I picked up a book about wildlife which was written in Afrikaans and I started translating it to English on the fly, out loud.
For those that don’t know, modern English decends from Old English with tons of Norse, Latin and French tossed in. This is why English seems to lack a coherent set of rules… there’s nothing but exceptions to the rules it might otherwise have because all the languages it’s based on have different rules. Old English itself is largely descended from Dutch, or a Dutch relative and a surprising number of the words they use are pronounced identically (cheese, bread, meat, beer, blue, etc…) but are spelled in a way that would make Chaucer giggle. Once you figure out how they use vowels and some odd uses of the letter “g” though translating the language to English is almost unnecessary as it’s so much the same as English and the rest you can get from context pretty well. For someone that truly sucks at languages, this was a nice experience.
If you’re thinking of a trip remember that you don’t have to know a lick of Afrikaans to get along. Everyone there, EVERYONE, speaks English pretty fluently if with various accents and sets of commonly used phraseology. All the signs are in English. The retail infrastructure is quite a lot like a mix of European and American. It lacks the number of big box stores and has oodles of smaller more specialized shops (though big box stores are there).
It’s almost like visiting San Diego in a lot of ways, especially in the visual appearance of the landscape. Apart from the racial makeup (who is in the minority) you’d have trouble figuring out you’re not in a city in the American southwest somewhere, other than all the cars are diesel powered and they drive on the other side of the road.
Also, for your first time especially if you’re an American: DO NOT RENT A CAR TO DRIVE THERE. You really need a primer trip where you get driven around first so you can see how the locals actually drive or you most definitely will find yourself in a surprising situation or two and may piss someone right off by trying to be safe instead of polite. If you’ve driven in India you’re probably not going to be surprised as much but otherwise, take the first one with a local driving you around.
Knife made for me by Danie Joubert.
.308 150gn Sierra Game Kings working springbok innards.
Draining blood and guts from my eland.
One of the two toughest little girls I’ve ever met.
My eland on the ground.
Tracking my eland as it walks off to die.
Rough country doesn’t mean you can’t prone out properly.
Filling my guide in on where the eland went to die.
Getting our stalk on.
Africa is as austere a place as it is beautiful.
The tall thin trees are actually aloe’s. 3m tall aloe. Just wow. Never knew they grew into trees!
Authentic South African Boer recipes. In Afrikaans of course.
A braai of livers, hearts and sausages. OMG that was good.
My springboks getting their cool down on.
That’s a happy hunter.
With a poker face carved in stone.
My trophy springbok.
My first springbok.
A landscape that says, “You’re totally alone here. Don’t get injured.”
Outline for a little classroom ballistics session.
It wasn’t a formal class, actually greatly condensed. Still, we wanted everything to look nice.
Sakkie and his bride. A more amazing pair of honest, hard working, kind and generous people I have not met. Only James Yeager comes close (if you haven’t met James, you don’t know him).
It’s winter in Africa. And quite cold and windy.
Landowner warthog trophy.
Landowner bushbuck & caracal trophies.
A caracal trophy.
My impala ram.
My impala ewe.
The buckey (what South Africans call a pickup truck)
Jacques confirming zero.
Black giraffe not 100m from our accommodations.
Johannesburg, SA suburbs. They name restaurants very strangely there.
Teaching ballistics eventually leads to a range session.
Dubai looks cool from the air.
Russia looks greener than I ever imagined it.
Teaching a little ballistics to a few friends.
Cruising through acacia trees on dirt roads. Like being a kid again.
These guys are great people, great shots and great students.
A landscape you just can’t find anymore. Empty of apparent human activity.
I just got back from South Africa and thought it might be nice if my followers got to see some highlights before I get into the weeds with longer articles on smaller subjects.
In a continuation of our re-launch from the close call earlier this year, we’re launching our new logo. This new logo will begin adorning the website and all new versions of BallisticXLR, BallisticPRS, BallisticRexLR and our latest product being announced separately. Big thanks to my ever present partner in crime Wouter for crafting this killer logo.
Remember: Don’t run. You’ll only die tired.