On the MMO I’m working on, we do quite a bit of service monitoring via jetty. Things like having a URL that reports back version information for a component, or a page full of statistics showing recent activity, or a way to trigger a self-check in a component to determine if the component is sane and healthy or not.
For certain components we run multiple instances of the service on the same box. In development environments, it’s a pretty small number of instances. On a production environment it’s a much bigger number. Typical stuff. However Operations noticed that startup times for a server full of instances jumped from seconds to over 20 minutes when the number of instances was increased from 3 to 12. Ouch.
So, something is blocking on startup. After some digging, I came across this post explaining how jetty uses a secure random number generator for session ids, which is based on the pool of entropy generated by the system. Sure enough…
$ cat /proc/sys/kernel/random/entropy_avail
(On development cluster machines that value is upwards of 3000)
Since we don’t care if the jetty session IDs are securely generated or not, switching the generator from secure to not-secure took us back to a few seconds to start up 20 instances.
Monday, June 27, 2011
Tuesday, June 21, 2011
Based on how much candidates and interviewers both seem to enjoy this, I hardly see any reason to do a whiteboard interview at all anymore. Personally, I don't even look a resume before I do a pairing exercise with a candidate. I don't feel the need to know anything about someone going into it, and I feel like I learn all I need to spending two hours with them on a task.
If you're having trouble hiring good programmers, give it a shot. Replace part of your interview process with a pairing exercise and see if it makes a difference.
Sunday, June 19, 2011
What if we turn the problem on its head: instead of storing data and then executing batch queries over it, what if we persisted the query and ran the data through it? That is the core idea and insight behind Event Stream Processing (ESP) systems: store queries not data, process each event in real-time, and emit results when some query criteria is met.
Thursday, June 16, 2011
Wednesday, June 15, 2011
Monday, June 13, 2011
People who work in software are smart people who take pride in their abilities to understand complex information and solve difficult problems. But much of the work isn’t only about smarts. Creating most software requires the help and cooperation of other people. Telling, convincing, and winning arguments won’t work to bring people along, change their minds, or help them help you. That requires influence.
Thursday, June 9, 2011
Tuesday, June 7, 2011
Bob may have the best ideas on the team. We don’t really know if that’s the case. No one else’s ideas are fully considered. We do know he doesn’t have a perfect record. Some of his fixes don’t work the first time. Some of his fixes break something else. If the team had a process to consider and refine ideas, that might not happen as much.
It may sound like it will take more time to separate generating ideas from explaining, exploring, and evaluating them. It may seem like a lot of effort to find more than one idea and test the ideas for soundness and test the level of support for a given idea.
But in years of observing teams, I find that slowing down and separating the steps of the choosing a solution helps the team speed up. A mashup process forced by a dominant individual may appear to save time in the very short-term. That’s seldom true if you account for all the time costs and other effects incurred.