Guest Blog: A Tester's Gotta' Know His Limitations, Domain Knowledge Really Does Matter

Insight Main Image

When I first came into the software testing world, it was from a decidedly indirect road. I was planning on becoming a musician, and I was pouring the majority of my energies and my efforts into that process during the late 80s and early 90s. So much so, that I tended to see the world through that lens. Everything was music, or performance, or signal processing. Sound is a neat analog to a lot of things; if you can understand sound, and the things you can do with it, you can carry that knowledge into a lot of places.

When I started my first real job, it came about because of a happy accident. I was working for a temporary agency that sent me to a small networking company. They were growing fast, with so much work to be done that they were open to anyone with drive and ambition coming in to do it. I had a chance to talk to a number of the testers, and they showed me their lab. As I walked through and looked at their setup, I saw devices that I would later learn were Ethernet and token ring hubs, serial connection boxes, routers and, along the walls on various desks and benches, computers of various makes and models. There were PCs, Macs, UNIX workstations, a DEC VMS minicomputer, and a few other exotic devices. The part that got my attention, though, was the bank of patch panels in the middle of the entire setup. Patch panels are a key and critical part of wiring up a recording studio. Without them, you have to unplug and replug a lot of devices in different ways. With a patch panel, you can easily make connections as you need them, and then break them down when you don’t.

Why do I mention this? It’s because that one simple insight was enough to help me explain how I could help this testing team get a handle on their test lab. I could rewire it, I could make it more efficient. I could make the setup more robust and easier to configure. Why did I think I could do that? Because I knew how to wire mixing boards and outboard effects. I understood the basics of signal processing, and how devices, if they are chained over too great a distance, can degrade sound or introduce noise into the mix. I had knowledge of a particular domain (signal processing from a sound studio and a musician’s perspective) and I also knew a bit about how to configure those devices through their small one inch console screens. Could I leverage those same ideas into designing a networking lab? Understand I had no real knowledge of networking. It was a gamble, but I figured I had little to lose, so why not go with it?

Because of that experience, and a lab administrator willing to indulge me, I set about learning all of the equipment in the lab, and how it fit together. What would it take to make end to end connections for all of the machines? What needed to be in the middle? What devices did I have to learn how to configure? What came down to just wiring? Could the patch panels really be a key to unlock this lab, and its domain, so that I could be effective? The answer proved to be yes.

Over the ensuing couple of years, I learned how networking equipment was similar to, and different from, sound equipment. I saw how ideas such as switching and routing of packets, and the use of various networking protocols, would let me set up networks so that Macs running Appletalk could connect on the same networks as PC’s running TCP/IP, or Novell, or Banyan Vines. I learned how to configure a router to act as a firewall so that some traffic could come in, but other traffic could not, as well as how to contain all but certain connections to go outside of the lab. I learned how to make simulations of multiple network segments connecting together, and using routers and switches to enable traffic to flow from one network to another.

It was heady and fun to work in this environment. Over the course of two years, I got pretty good at being able to configure a broad array of networking setups. As the company grew, and the testing team had a need for more people to do even more testing work, I was asked to join the team as a full time software tester. Thus, over the next seven years, I would get to know small sections of our product at a given time. I would learn about the route processor and what it could do, and what needed to be tested. I would learn about interface cards, and how they were expected to behave under normal and adverse conditions. I would learn about network management protocols like SNMP, and how to manipulate MIBs to get information from systems. I would learn about content delivery devices and caching, and how to measure the load time of various web pages and content with or without these devices.

Networking and sound systems had a small amount in common, but there was enough that I could leverage and use to my advantage. I came to believe that I could test anything after these experiences, that I had a knack for testing, that I understood how things worked, and that I could carry those ideas anywhere. I developed a bit of “hubris” regarding my skills. Hubris is a Greek word that means “a pride in one’s own accomplishments or skills.” Often, it’s not wholly warranted. The ancient Greek philosophers and storytellers often paired hubris with another term, “nemesis.” Nemesis is punishment or retribution meted out to those who commit hubris. I would find out soon enough just how appropriate those terms were.

After I had been with the networking company for a decade, I decided to see what I could do in other environments. I felt confident I could test any software project or product. That is, until I found myself working for a company that made capacitance touch devices. What are those? If you have ever used a touch screen, or a touch pad, or a little eraser sized rubber circle in the middle of your keyboard to navigate, chances are, your using a capacitance touch device. I went to work for a company that was actively involved in developing prototypes and models of these devices for large computer manufacturers. Networking was not very important here. The laws of physics were king. I will hereby confess that my knowledge of physics was woefully inadequate. Still, I believed that I would be able to leverage my testing skills and my previous experiences, and I’d just pick up the rest of the details along the way. While I was partially right (I was able to learn quite a few things, and I was able to test somewhat effectively) there was one big gap that, try as I might, I couldn’t effectively overcome. I saw that there were unusual spikes and variations when I subjected a device to a shaker table test, but I couldn’t effectively communicate the data I was receiving, or present it in a way that was useful. I found myself relying on the hardware engineers to write little tools for me to run the resulting data through and create reports from the data collected from long term vibration runs, deep freezes, extreme heat, pressure tests or utilizing electrostatic discharge. While I could set up those tests, what I couldn’t do was effectively synthesize the information I was receiving, not without a lot of help.

Over time, this became obvious to the director of our group, and I was called into his office one day. When a discussion starts with:

“I’ve been looking over your resume, and you are not a hardware engineer. You’re a software tester who worked on networking equipment. How did you get hired here?”

Well, the writing was on the wall. Needless to say, I found myself looking for other opportunities. My hubris had met its nemesis.

This was a golden lesson to me. It taught me how domain knowledge is often the secret sauce that makes the difference between being just a good tester and being a great tester. Knowing the fundamentals about how to test is one part. Knowing how software is created and understanding the logic of systems is another. The most important aspect, though, is the in-depth knowledge of the field that you are working in. Often, you cannot make a leap from one field to another without preparation or exposure to that second environment over an extended period. I came into the networking world with a relatively simple skill. A kid with some peripheral experience wiring up recording studios was able to leverage those skills as a lab administrator. Over time, I also slowly learned the world of computer networking, as well as software testing and what made for good testing from my peers. The domain knowledge didn’t just come in one fell swoop; it came because I worked with it daily. I wasn’t expected to know networking, but I learned it over time. In the job where I was testing the laws of physics, I was expected to know all that, and I didn’t.

So what can we do if we want to work in an environment where we don’t have the domain knowledge to be effective? How do we gain it?

First, ask yourself why you want to be involved in a particular industry. If you are really interested in a given field, do all you can to learn about it. If you want to work in an area where chemical engineering is an important part of the job, then you will need to go and learn about chemical engineering, which will likely mean you will need to understand peripheral disciplines, such as chemistry, biology, geology, and mathematics. Determine if you are willing to go as deep into those fields as is necessary to be able to speak in a meaningful way about those topics. If you are looking to test a web site that is dealing with very large data collection, then you will need to know how massive databases are created and optimized. A book or two might be all you need to get up to speed enough to be effective. If you want to learn about a technical topic in a fun and interesting way, I’d suggest looking at “The Manga Guide” to series of books offered by NoStarch Press. These take what can be challenging topics, and help break them down in a way that makes them accessible and understandable to a broad audience. They should not be your only source of information, but they can certainly be an excellent start (I wish “The Manga Guide to Physics” was available to me in 2002).

Second, ask others, who already work in the area you are interested in, what they do every day, and what they are expected to know. If you want to work with testing small devices like mobile phones, you may find that you are expected to know a bit about electronics, and how to do basic rework on a circuit board. I don’t mean design or fabrication, but you may find you need to know how to solder a lead wire from one point to another on a very small surface. Yes, software testers may be asked to solder occasionally. That is a domain specific skill. It helps to know when you are going in to a job that requires it if you have a steady hand or not. If you don’t, know that going in, and be honest. If you don’t want to let that stop you, then invest in some simple electronic kits and practice soldering for awhile until you can do it smoothly.

Third, read about the greater trends going on in your industry and determine what publications are available, or what web sites can be accessed that deal with these given fields. Right now, I am working in an environment where a primary customer of our product is the U.S. Government. Because of that, we have special requirements we have to meet with regards to accessibility. This refers to the ability to use peripheral devices and software so that people with disabilities related to vision, auditory or tactile interaction can still use our products. There’s an entire section of a law that is dedicated to this. It’s called the Section 508 Amendment to the Rehabilitation Act, and yes, for my given scope and work, I need to be familiar with it and what it means for our products to be compliant with it. Thus, it falls to me to learn these regulations and put them into practice. You may find that similar needs exist in the area you are interested in working in.

Domain knowledge is, ultimately, the make or break item that determines if you are successful. Testing techniques can be learned and refined with practice and effort. Coding knowledge can likewise be learned and practiced, and expanded over time. If, however, you are deficient in the domain knowledge of your chosen area of testing, no amount of coding or testing skills will overcome it. The good news is that you really can learn about any endeavor if you really want to. Some areas will be more difficult than others. I wouldn’t recommend applying for a job at a biotech company if your background doesn’t already include a fair amount of bio science. Still, if you are determined, and you have a genuine interest, there’s a lot of information that’s just a mouse click (and some steady reading and practice) away.

Related Articles: