We gladly bring to the public one more interview, this time we interviewed the surprising Jack Ganssle. Jack, for those who don’t know him, is one of the leaders of the quality movement in embedded systems. In this interview Jack gave us very sensible and correct information about embedded systems development. He shows us it is not required big changes to get big increment in quality, but with a few changes in the right places it is possible to get significant improvements on team throughput and less bugs on products. This is why Jack Ganssle was surprising to us, he showed that achieving great results can be simple.
The interview has been conducted by our show man André Prado, and the questions and transcription were made by André Prado, as well, by Henrique Rossi, Marcelo Jo, Thiago Lima and myself. Below follow Jacks’ words and I hope that they are for the reader as illuminating as it was for the Embarcados Team.
André: Why did you choose embedded systems?
Jack Ganssle: That’s an interesting question, I sort of fell into it. Back in 1971 I was working as an electronics technician while I was in college and the company where I was working started doing one of the very first embedded project using the first 8 bit microprocessor, the 8008, and no one knew anything about computers so they promoted me engineer because I knew about computers. That was 40 years ago and I just have never left the field. To me it’s fascinating, because we do the hardware and the software. I know a lot of people getting into just the software side, that’s great, but for me the exciting part is being able to do both these things.
André: So you do hardware as well, are you used to work with FPGA or ASICs?
Jack Ganssle: Oh, yeah, a lot of it, FPGA is so common in the embedded world, and becoming more so. As a matter of fact, I just started playing with this Xilinx Zynq board, it has two Cortex-A9 processors plus a bunch of FPGA fabric. It’s way cool, so I was playing with it just this week, actually.
It used to be that if you wanted to do something custom you’d build an ASIC, but who can afford to do that anymore? 20nm is something that costs a couple of hundred million of dollars to do an ASIC. You can get a 5 dollar FPGA, so for high speed work an FPGA is definitely the way to go.
André: What are the most frequent reasons companies seek for your help?
Jack Ganssle: They are looking for two things: to get their products out cheaper and, probably to a lesser extent, they want of a higher quality. For management the idea of getting done faster is what drives them. The funny thing is, as I keep saying in my seminars, if you do it right it will be cheaper in the long run. We have tons of data that shows us that the average firmware project spends half the schedule doing debug. That’s kind of crazy, they spend half of the time fixing their mistakes. We know that the best way to do it is just don’t make those mistakes in the first place. I look at it this way: if half of the schedule is debugging, the other half must be called bugging.
André: Jack, even on current days, a huge amount of companies doesn’t mind about quality, instead they want their projects done now, what do you think about that?
Jack Ganssle: Quality is my pet peeve, we just don’t do the things we should be doing to get the quality needed. Toyota had to pay 1.2 billion dollars because of their bad software, and if you think about it, that code is now the most expensive software anyone has ever written. It is true we all need to rush but it will always cost you more. There is a quote that I really like: shortcut makes for a long detours. In the past we could get away with it because our systems were quite simple, but they are a lot more complicated now, so I don’t think we can continue doing things without quality. And the thing is, we know how to do it right, and we choose not to do so. To me that is more like a mortal sin, it is no longer acceptable. We’ve had 30 years to teach us the right way to build embedded systems, and, sooner or later, people are going to start getting really hurt with driverless cars, avionics and even medical instruments. There was a big recall in US a few years ago of infusion pumps, these machines used to inject drug in the blood, due to firmware errors and other problems.
Mostly companies I go to have embraced the quality for manufacturing because they discovered if they do high quality manufacturing they save money. But very few really accepted quality on the engineering side, especially for firmware. I’m starting now to see more interest, because people are starting to actually see the problems that are costing companies lots of money. I’m hopeful that its going to change, but it is happening really slowly.
As you said, companies want the project done now, and if we build a million lines in a month, sooner they will want it done in two weeks because of competitive pressures. At some point this has to stop, as it is unsustainable. The old all saying is, if something is unsustainable it’s going to stop.
I see a lot of promise with the younger engineers, because they haven’t been spoiled by decades of doing it wrong. I know a couple of American companies that only hire young engineers, right out of the university, so they can teach their way of doing things.
André: What the biggest obstacles when installing quality in a company, maybe convincing the management?
Jack Ganssle: It’s true we can blame management, but I think that a lot of times, we engineers point the finger at the management, when we should be pointing at ourselves. In many cases we are very willing to take shortcuts, because writing code is fun, debugging is fun too, but sitting there doing design and getting requirements right, for a lot of people, it is a drag. So we skip over that phase and jump right into coding. It’s like going to a machinist’s shop and asking him “I want you to build me an engine” without giving him any plans. He will just start drilling holes in pieces of metal and it would take him forever to finish, but if you give him plans, detailed and correct plans, he’s going to get it done in no time. That’s where creativity really is, in the design, and, for me, I get much more satisfaction with the design, and then I can turn it over to people do the implementation.
André: Do you focus on processes like TDD or agile ?
Jack Ganssle: Yes, I focus on a lot of processes, but I’m not a fan of TDD. When I see it used it is practically hacking. People are just writing as much code as fast as they can and they are not making sure that the design is decent. How do you really know that the tests are correct? Do you know James Grenning? He is one of the guys that signed agile manifesto. He teaches TDD in the embedded world and he is a great guy and a great friend, but we disagree on this. We have a lot of fun discussions about the subject. I think agile people get a lot of things really right, the focus on spiral process is right, and they really believe on extensive testing which is so important, So they say a lot of great stuff, but often they deprecate design, which is an increasing problem especially if systems get more complicated. So, yes, I talk about processes issues without thinking in terms of heavy process, but more things like what we can do in a spiral fashion to develop a product. I talk about metrics. You know, we are engineers, and engineers are numbers people. But if you ask a firmware person about practically any metric it’s hard to get answer. Like, “what’s your bug rate?” That’s a critical piece of information that you can measure in a million different ways, but mostly people don’t know anything about what these numbers are. “What is your real time response, if you’re writing an interrupt routine?” In other words, does that execute in a microsecond, or in a millisecond, or a week? People don’t really know. I think that is a significant problem, yet metrics are so easy to get. As an engineer you design something, you predict what’s going to happen, you implement it and you measure what you did to make sure that what’s really happening matches what your predictions are, If they are off, then you have to ask the hard questions like “why are they off?” Do I have a design error? Did I make a mistake?
André: So, you don’t agree with the agile movement? It is so fashion now a days to say “I do agile”.
Jack Ganssle: In software there are a lot of fads. There’s always a fad. Almost every one of these fads has had great things to say and it improved our body of knowledge. Agile has fantastic ideas, it is a wonderful thing that has happened to software engineering, but, almost every company that I go to where they say they practice agile is really just a cover for chaos. They say “oh, we’re agile, you can’t ask us about schedules, we are agile”. To do agile correctly is a very disciplined process, but it is pretty unusual to see such a discipline.
André: What kind of metrics do you usually consider in a team?
Jack Ganssle: There are a bunch of those, we can touch on a couple. I think it critical to know how many bugs you injected into the system, and what percentage were removed before you shipped. We hope it is a hundred percent, but it probably isn’t. There are published figures on how highly disciplined places do and what terrible companies do, and if you don’t know where you stand, you don’t know if you are doing well or doing malpractice. And is the saying goes, you manage what you measure. If you won’t measure anything, like bugs, you won’t manage bugs. There are interesting data that shows if you measure just those two things, you can get order of magnitude reduction in bugs over the course of a year or two. I also believe in measuring everything about real time performance. A lot of people say it is hard, and it might be, but with an oscilloscope and a few things you can get data really easily. There are some awesome tools like uC/Probe from Micrum, and the Threadx people have some great profiling tools. If we don’t get this kind of information we really don’t know where we stand.
André: Can you tell a very successful case as well a very unsuccessful case of embedded systems?
Jack Ganssle: I’ll tell you the most successful embedded project ever, it was the space shuttle. They averaged one bug for every 400.000 lines of code. That is incredible, they did a fantastic job. There’s another one called Tokeneer, a government project that was done in a language called Spark. (Spark is a subset of Ada with some proof checking extensions). They had one bug, just one bug, in the whole project. I saw a military airplane avionics project done in Spark; they ran all their proof checking tools and fixed whatever the tools found. When it was turned on there were zero bugs – they never found a bug! I know some Spark people who have forgotten how to use their debugger. Most of us will not going to get there, but we can do whole lot better than we are. The average embedded project runs 5 to 10% error rate after a clean compile, so if you have written 1000 lines of code there are 50 to 100 bugs on it. That’s a lot. To reduce that by an order of magnitude is just not that hard. We know we should be using standards to write the code, doing inspections, and doing other practices. But most of us don’t want to be bothered. Sure, management has their share of blame, but so do we.
André: And what about the failure case?
Jack Ganssle: There are so many to pick from, there is the Therac 25, a device that killed cancer by shining radiation at the tumors. It actually killed people because of software problems. There was another similar machine in Panama, 10 years ago, that killed 21 people because of software problem. It is very sad. If you actually dig in to see what happened in these cases there was pure chaos. No one knew what was in the code, there were no real requirements, there were few design documents. They broke all the obvious rules and as result people were killed. There was a case where the British did avionics for a helicopter; after a very bad crash where 25 people were killed they reviewed the code. It was so terrible that they gave up trying to review it after they got through only 17% of it. That is sort of my message: we do know how to do embedded systems correctly, and if we are disciplined professionals we need to adopt best practices. I really believe we are going to, it’s just going to take some time.
André: Do you think new practices will be proposed or we are fine with today’s practices and just have to apply it?
Jack Ganssle: This is a cool field because there’s always new stuff coming out. One of the things I enjoy most is studying and keeping up with what’s going on, which is a lot of fun. So, new concepts and processes are coming out all the time, but I think for immediate gains we don’t need anything new, we just need to adopt the stuff that we’ve known for years and years. People say “Yeah, if you do that way you’re not going to get a 100% perfect solution” and they are right, but we might get 90%, which is an A grade. That’s a lot better than the Cs and Ds we’re getting now.
André: What do you think about the maker movement, like Raspberry Pi, Beagle Bone and Arduinos?
Jack Ganssle: It just happen I have a Raspberry Pi right here. I have mixed feeling about maker movement, It’s the best thing and worst thing. It is the best thing because these products get a lot of people interested in this technology. As an engineer, I get frustrated when people don’t care how things work, because it is cool and interesting. The maker movement is getting a lot of non techies into to technology and that’s is fantastic. My fear is, I see some who feel with a little knowledge they are electrical engineers. It is easy to make lights blink and do some cool things, but to actually make a real reliable product, to go from a thousand lines of code to hundred of thousands lines of code is much more difficult. Getting all that right requires an engineering mindset, I don’t know how this is in Brazil, but the press in US every time a 16 year old builds something with one of these boards, the press hails him as the next Albert Einstein. I admire the kid and I hope he does more of it, but the truth is, I hope he goes on and get a degree in engineering and really become a professional at it.
André: The embedded systems field in Brazil is small, so we don’t see these press much in Brazil. How do you see our embedded systems field?
Jack Ganssle: I think it is growing, I know a lot of companies are moving down to Brazil. All the semiconductor vendors are down there and what they are telling me they did get great engineers, less expensive than in US. The engineers have a great education, the timezone is very friendly to us and a lot of Brazilians speak English, which make it easy too.
André: What do you think of the offshore tendency to send the projects to be developed in India, Brazil, UK, etc. Do you think US might suffer from that or it is the expected procedure to maximise profits?
Jack Ganssle: I think it is all of those things. Companies are going to do it to maximise profits of course, but they are also doing it because they need the engineering resources. There’s a lot of debate is US over whether there is a shortage of engineers, but the fact is we have full employment. There’s very few engineers who don’t have a job. I have spent my life traveling all over the world and am finding great developers in all sorts of places. There are a million people doing outsourcing to India, for example. It seems that the most successful companies have their own operation there, in that way they can really shape the direction their projects are going. Brazil is a little different, because, like you said, it is a small industry there. But India is like Brazil in one way too, the engineers tend to be younger and very enthusiastic which I think is encouraging. I think overall outsourcing is going to be a good thing for the world. In some countries outsourcing is a way out of poverty and I think that’s pretty wonderful thing.
André: What about the quality of these outsourced projects ?
Jack Ganssle: I would say it is all about the same crap, it doesn’t matter where it is coming from. I’ll tell you a story though, I was in the Czech Republic last summer having a dinner meeting with some people. They told me they have a million lines of code in one project where all the comments are in Czech. Turns out there are only 10 million people in the world who speak Czech, so they have a real problem! When people do outsource they have to think about this. If the code is going to move around the world it probably should be documented in English.
André: Well, lets talk about you vision of the future, how do you see the Embedded Systems fields on 2020’s?
Jack Ganssle: Well, what I have seen over my career is that when you look 10 years on the future you project all kind of things, but when you get there what has happened is something you never would have dreamed of. For example, smartphones. 10 years ago no one would have dreamed that you’d have the internet in your pocket or you’d have a GPS in your pocket. When I was young it was illegal to own a telephone in the US. Now everyone has these amazing devices in their pocket, so I think predictions are hard. But I do think it is very clear that Moore’s law is going to do fairly well, for at least the next 8 to 10 years, which means in 10 years instead of having smartphones with 100 billion of transistors, you’re going to have trillions of transistors, and that’s is going to be a really smart smartphone. Robotics will be more important than anyone today understands. We are not far away from it today. The battery technology is getting better all the time – look at Tesla, they are doing great things with batteries. The mechanics is getting better, and the electronics is just about where it needs to be to make really clever robots. And, if you think about that, before long robots will be able to make robots, and robots will then be used to do all the dirty jobs, like mining, I worry that the value of labor will go to zero because robots will do all that work. Engineers will be in great position, because it will be a long time before robots are smart enough to do engineering, but I’m afraid for a lot other people. For example, with Google cars what will happen to taxi drivers when no driver is needed? So I think we are going to find that technology is going to continue to explode but we are going to see some really difficult social issues come up as a result of this, and I just don’t know what is going to happen about that.
André: How do you think quality is involved in that vision?
Jack Ganssle: Well, if you have a million cars driving themselves around the city, it better to be pretty good. When I go to Brazil I usually go to São Paulo and guess all I can say is that cars doesn’t move very fast there! But is interesting that the driverless cars are all going to be talking to each other, so you are going to have this extremely complicated network. Look at avionics – we know how to make complex systems work, so we will be able to draw on the lessons we’ve learned from designing avionics and other safety critical applications. There are amazing products available today to help, like highly-qualified Real-Time Operating Systems Greenhills’s, for example, has been proven to be mathematically perfect. So there’s some pretty cool directions that things are going.
André: What about drones?
Jack Ganssle: Drones is very interesting, because the technology is basically there today, I mean, it works, but it is raising all these issues like: Can you drive a drone over your neighbor’s property? Can you take pictures in your neighbor’s windows? It is scary, we’re starting to see these things and I think it is going to be a lot of interesting legal issues that have to be resolved for drones. Amazon is talking about delivering packages with drones and I think they are crazy, because the energy required to use a drone is so huge compared to using one truck that delivers to 500 houses in the neighbour, the energy budget doesn’t make sense, but for a lot of other applications we’re going to see a lot of drones out there, a lot more than we think, It’s going to be interesting to see what happens to the air space.
André: And the Internet of Things and Power Harvesting that is getting a lot of media lately?
Jack Ganssle: When I hear Internet of Things I want to throw up. It’s just a marketing buzzword. We’ve been doing the Internet of Things for 20 years. We’ve been collecting data to the Internet for years. So this is just a new word that the marketing people are using, but the fact is, because of some newish technology, like the ARM Cortex-M0, you can buy a 32 bit microcontroller for USD0.50. Most have Ethernet and Wifi on board, so we are going to see a whole lot more stuff connect to the internet. It’s getting easier and easier, because we can get open source TCP stacks. And that’s the other interesting thing, a lot of these small MCUs takes almost no power, they can run on just milliamps, so you can harvest power from all kind of sources. Power Harvesting is going to be a huge area, because batteries are relatively bulky, need charging, and are expensive. For a lot of applications energy harvesting is going to be the way to go. Supercapacitors have got so good that is easy to energy harvest and charge a supercap. I keep thinking about a legend we have in US, about Johnny Appleseed, where Johnny spreads the apple seeds as he walks around. That’s what we are going to have with sensors; they will have self-assembling mesh networks that will connect to wifi, GPRS or one of those things. There will also be some social issues, like with robotics, but I think smart sensors have tremendous benefits.
André: What do you think of legacy technology such as 8 and 16 bit microcontrollers?
Jack Ganssle: Well, it’s complicated, you know, what if you could build a system for a hundredth of a penny, then you could do the Johnny Appleseed thing, so maybe there is a compelling market for 8 bits. These processors are all made in obsolete technology like 250 or 180nm, and all those fabs have all been paid off. If you do something in 45nm fab there is still a couple of billion of dollars that has to be amortized on the price of the products. And with ARM there’s always the ARM taxes, because you have to pay to ARM Holdings a certain fee every time you sell a microprocessor. So, 8 bits are always going to be cheaper. There is certainly a huge shift to 32 bits, there is no questions about it, and 32 bit will take over more and more of the market, except for the narrow niches, where cost or power matters more than everything else. Some of the vendors of 8 bits chips tell me that the reason that they still have such a good market is because a lot of their users aren’t firmware people. They’re just going to write a couple of hundred lines of C, they don’t want to use something complicated, just want to use a PIC something because that just a simple thing to use. That’s kind of interesting thing too. ARM parts have really complex I/O usually, which many people won’t want to master.
André: Do you think the future is all about ARM?
Jack Ganssle: I think the near future is all about ARM. Intel really wants to be in the embedded space. They keeping missing it but they have some really smart people, so I suspect that they will figure it out. But, for next 5 years ARM will be probably 60 to 70% of the embedded space. After that, who knows, there could be some outrageous inventions we can’t even begin to dream of right now. As a matter of fact, I’m sure that there will be.
André: What do you think about the FPGA usage in industry, some sources of public research claims that FPGA use is decreasing over the years, while the intention of using it, according to them, is increasing?
Jack Ganssle: Yes, I participated in helping to construct some of these studies, and what I find is the questions are typically poorly put together. If you look at the data that comes back from the surveys, you just can’t trust it. You’ll have someone who says “I am a firmware engineer, I don’t do any hardware, but I design FPGAs”. Sometimes the answers don’t make any sense. The market is increasing, Xilinx and Altera get bigger every year, so I think FPGAs use is definitely going up. There’s a Xilinx part which will go in rearview mirrors in cars, with cameras, to watch the traffic to see if the car is getting out of lane. That’s tough thing to do with software. Image processing is hard to do with code because of the data rate is so high, but for FPGA it’s natural, and I guess what I’ve observed over my years in this field is that data rates go up every year. I got a new digital camera not long ago with a 20 mega-pixel sensor. The amount of data coming out of this thing is crazy. Processors can always get faster, but they are bound by memory, and memory has not gotten any faster in years, so doesn’t matter how fast the processor is. FPGA can do a lot of that stuff in hardware really fast, so I am very optimistic about that market.
André: What about your experience in Brazil, did you have the opportunity to consult here?
Jack Ganssle: First time I went to Brazil was in 1978, for business, I was down there for a month working on a bunch of machines, and went all over, Porto Alegre, São Paulo and out in the jungle, and it was fantastic time. Now my trips are been very short, for a couple of days,just to talk at conferences, or to do a little consulting for a day or two. I’m not real fan of São Paulo because the traffic is so brutal, but the rest of the Brazil is fantastic. I love the Brazilian people, as they just so much fun and generally nice. Caipirinha is one of the reasons I am disappointed that I’ll not be coming down to the ESC this year!
André: What advice would you give us embedded engineers to perform better on our jobs?
Jack Ganssle: Well, that’s actually a great question. I always tells young engineers the first thing they should do is save money, save a little money out of each paycheck, you get older faster than you can possible imagine and someone that has money in the bank has options. If you don’t have money in the bank than you have to scramble for a job. Second thing I always say is engage with the field, always keep learning. This is a cool field and there’s a lot of great stuff going on, but I see a lot of engineers who pretty much stop any form of formal learning when they graduate from college. If you going to be in this field, you have to keep studying for your whole career. And the other thing I say, this is an advice given to me by a friend many years ago: you have to make money, and you have to have fun, either one alone is not enough. Often times we just think about the money but you going to be doing it for 40 years, so you want to make sure you have a big smile in your face when you get up in the morning.
André: Do you still have fun coding or design is more your thing?
Jack Ganssle: I don’t do that much coding, although I was doing some just when you called. I find its more fun to look at the overall system architecture. I don’t really do much consulting the way people think of consulting, but I often time give companies architectural design advice. Occasionally I’ll go in solvable problem for a company because I like being in hardware and software side of things. I was working in Sweden for a few days a couple of months ago, they had a problem with 50ps timing error, which was very interesting to chase down, but I don’t like to sit down and work hundreds of thousands of lines of code anymore.
André: What do you expect from you work on the next years?
Jack Ganssle: Sometimes my wife says I should retire, but I’m having too much fun. You know, I don’t have a real job, I do lectures, a little consulting and a few other things of that nature, a lot of writing. It gives me tremendous amount of freedom and I enjoy that a lot. And I do my sort of projects and experiments. Recently I’ve been experimenting on how small batteries behave in very low power systems, just out of curiosity. I see myself doing this for maybe more 10 years or so, I hope.