After the HTS application was used throughout manufacturing, we begun to notice a series of periodic test failures that could not be traced back to a specific component failure. As part of the Lean manufacturing initiative, these failures were labeled “No Problem Found” (NPF), and they were set aside for investigation into the root cause of the failures. I lead the team that was responsible for investigating the root cause of all of the NPF failures, and returning the inventory back to production after the root cause was solved. The largest obstacle to this project was the small numbers of failures and the difficulty of reproducing them. This project became part of the CAPA process for manufacturing.
To dig down to the root cause, first we collected all of the failure information from the HTS database. This was ranked, in order by the number of occurrences. Then, we selected the NPF with the highest number of occurrences and began the investigation into the root cause of the test failure.
Most of the changes to the HTS test system were to improve test timing, work around a product software flaw, or to implement new test requirements. When changing a test requirement, we completed statistical analysis of the test results for the biological parameter in question, designed new limits for the parameters, and reviewed the changes with the design engineers before making the changes.
When we made updates to the HTS test system, we discovered that the failures were usually due to a race condition in the test software, and that we could make slight timing changes to the timing of the tests to resolve the problem. We would do this after analyzing the statistical distribution of the timing of the hardware system under test.
Over a period of two years, the team determined the root cause of 28 different NPF issues. We returned over 1000 units back from the NPF area back into production. The rate of NPF issues was reduces to less that 1% in the Dash 3000 HLA area.
This project increased the productivity for High-Level Assembly area, and we saved ~$64K in cost avoidance in 2003 and $46,000 in cost avoidance in 2004, by moving the solutions to other Acquisition products. The cost reduction comes from the removal of unnecessary rework and retest costs in manufacturing.
Sunday, June 28, 2009
Life Stories #6 Hardware Test System (HTS) Architecture
I began working for a medical equipment manufacturer in 1997 as a Software Engineer in operations. At that time, the company formed a new test engineering team to architect, design and develop an automated test system for testing patient monitors in the manufacturing environment. The goal was to design a software system that could test multiple patient monitors at the same time, and store all of the results to a central SQL database. Each patient monitor has multiple biological parameters that it monitored simultaneously. The test system needed to run multiple tests on a patient monitor simultaneously. Because the common behavior of all of the patient monitoring equipment, we could leverage economies of scale by using common test software, but varying test parameters, to test the patient monitoring hardware. The central SQL database would hold the test parameters; also, it would hold the relationships between the device under test (DUT) and the tests for that device and the test parameters for each test for that device.
This was a large system and we designed it using knowledge of past systems, but without using any components (hardware or software) of past systems. There were two major components: a custom hardware simulator for simulating all of the biomedical parameters (BP, Temp, ECG, SpO2, NBP, IBP, EtCO2), and a Windows NT/XP software application for running the tests on multiple units simultaneously. The largest obstacle was designing the architecture correctly, so it would be flexible enough to allow for the addition of new tests, simulators and test targets (DUTs).
We decided to use the Rational Unified Process, the Unified Modeling Language, and Rational Rose to do the Object-Oriented Analysis and Design. We analyzed stakeholder needs to develop Use Cases. We created use case diagrams to show relationships between the actors and the use cases. We developed several analysis models of the system using sequence diagrams, activity diagrams and analysis classes. We developed the overall architecture with other project members using UML and analysis modeling techniques.
We developed the overall architecture by printing a massive collaboration diagram from all of the use cases and then grouping the logical classes into groups. We drew a border around each of these groups; these became the packages of the system. Then, we designated the classes that would become the interface into each package, and the classes that would define the framework for each package. Each package became either a Dynamic Link Library or the application executable.
Next, from the use case analysis, I design the interfaces for all of the systems for the entire architecture and the test engine for the test system. Then, we divided the software development work among the various members of the team. I handled the GUI, the DLL that communicated with the patient simulators and the patient monitors, the DLL that held the test engine, and the DLL that held the concrete test for the DUT. The other team members in our group worked on the DLL that communicated with the Oracle SQL database and the web server software that read the SQL database and generated reports for use in production.
We wrote all of the software for HTS using Visual C++ and the MFC library. In addition, I developed multiple COM components for use with the Hardware Test System. All of these COM components were created using ATL.
This test system became the flagship test system for our corporation. It is now used in multiple facilities around the world. It proved to be flexible. We expanded it to test multiple types of patient monitoring equipment and telemetry equipment. We included the use of HP signal generator and spectrum analyzers. All of this was done without making major changes to the software architecture.
This test system now tests nearly the entire patient monitoring equipment portfolio that the company manufacturers. It is difficult to estimate the amount of cost savings by using the HTS, but it must be more than 1 million dollars to date. This project started in 1998 and the system is still in use today.
This was a large system and we designed it using knowledge of past systems, but without using any components (hardware or software) of past systems. There were two major components: a custom hardware simulator for simulating all of the biomedical parameters (BP, Temp, ECG, SpO2, NBP, IBP, EtCO2), and a Windows NT/XP software application for running the tests on multiple units simultaneously. The largest obstacle was designing the architecture correctly, so it would be flexible enough to allow for the addition of new tests, simulators and test targets (DUTs).
We decided to use the Rational Unified Process, the Unified Modeling Language, and Rational Rose to do the Object-Oriented Analysis and Design. We analyzed stakeholder needs to develop Use Cases. We created use case diagrams to show relationships between the actors and the use cases. We developed several analysis models of the system using sequence diagrams, activity diagrams and analysis classes. We developed the overall architecture with other project members using UML and analysis modeling techniques.
We developed the overall architecture by printing a massive collaboration diagram from all of the use cases and then grouping the logical classes into groups. We drew a border around each of these groups; these became the packages of the system. Then, we designated the classes that would become the interface into each package, and the classes that would define the framework for each package. Each package became either a Dynamic Link Library or the application executable.
Next, from the use case analysis, I design the interfaces for all of the systems for the entire architecture and the test engine for the test system. Then, we divided the software development work among the various members of the team. I handled the GUI, the DLL that communicated with the patient simulators and the patient monitors, the DLL that held the test engine, and the DLL that held the concrete test for the DUT. The other team members in our group worked on the DLL that communicated with the Oracle SQL database and the web server software that read the SQL database and generated reports for use in production.
We wrote all of the software for HTS using Visual C++ and the MFC library. In addition, I developed multiple COM components for use with the Hardware Test System. All of these COM components were created using ATL.
This test system became the flagship test system for our corporation. It is now used in multiple facilities around the world. It proved to be flexible. We expanded it to test multiple types of patient monitoring equipment and telemetry equipment. We included the use of HP signal generator and spectrum analyzers. All of this was done without making major changes to the software architecture.
This test system now tests nearly the entire patient monitoring equipment portfolio that the company manufacturers. It is difficult to estimate the amount of cost savings by using the HTS, but it must be more than 1 million dollars to date. This project started in 1998 and the system is still in use today.
Friday, June 26, 2009
Life Stories #5 Fixing Sway Bar Links
Recently, I sent our car to the mechanic for some general maintenance: oil change, transmission fluid change, differential fluid change, etc.. During the inspection, the mechanic found some major issues (i.e., extremely worn ball joints and three broken sway bar links). The total repairs were near $1,000, so I decided to replace the sway bar links myself. This would save us about $200.
Once I got under the car, I quickly discovered that the nuts on the linkages were rusted on tight. Spraying WD40 did not help. Also, once side of the linkages is a round ball, which is difficult to hold onto with a pliers or a channel lock by hand. Finally, I did not have ramps or safety jacks, or a garage, so I needed to work on removing and replacing the linkages without lifting the car.
The first problem was how to hold onto the ball end of the bolts. I don’t have many custom tools in my toolbox, but I do have a pipe wrench that I used in the past for sum plumbing work. I figured that if a pipe wrench could turn a smooth, round steel pipe; it could hold onto the ball end of the bolt. This worked, but the heavy pipe wrench was difficult to hold under the car, lying on my back.
It took me about 2 hours to remove one of the bolts this way, using the pipe wrench and a normal wrench. I decided that this was way too long, and went and bought a hacksaw. Lying on my stomach, I cut through the rest of the bolts, using both hands to move the saw because of the limited area and range of movement. I was able to remove the rest of the bolts in only 2 more hours.
Finally, I was able to attach the new linkages to the rear of the car, saving about $150 and learning about the difficulties of removing rusted bolts. I still have the front driver's side linkage to fix, but I found out, via research on the web, that you can use heat and wax to remove the hard-to-reach bolts in the front of the car.
Once I got under the car, I quickly discovered that the nuts on the linkages were rusted on tight. Spraying WD40 did not help. Also, once side of the linkages is a round ball, which is difficult to hold onto with a pliers or a channel lock by hand. Finally, I did not have ramps or safety jacks, or a garage, so I needed to work on removing and replacing the linkages without lifting the car.
The first problem was how to hold onto the ball end of the bolts. I don’t have many custom tools in my toolbox, but I do have a pipe wrench that I used in the past for sum plumbing work. I figured that if a pipe wrench could turn a smooth, round steel pipe; it could hold onto the ball end of the bolt. This worked, but the heavy pipe wrench was difficult to hold under the car, lying on my back.
It took me about 2 hours to remove one of the bolts this way, using the pipe wrench and a normal wrench. I decided that this was way too long, and went and bought a hacksaw. Lying on my stomach, I cut through the rest of the bolts, using both hands to move the saw because of the limited area and range of movement. I was able to remove the rest of the bolts in only 2 more hours.
Finally, I was able to attach the new linkages to the rear of the car, saving about $150 and learning about the difficulties of removing rusted bolts. I still have the front driver's side linkage to fix, but I found out, via research on the web, that you can use heat and wax to remove the hard-to-reach bolts in the front of the car.
Tuesday, June 16, 2009
Life Stores #4 - Pacemaker Pulse Detection
At the start of my employment, I was given the task of taking an existing prototype pacemaker pulse algorithm and implementing it into a DSP to run in an embedded environment. Also, I was to improve on the already great performance of the algorithm.
Like most of the projects that I start, I was unfamiliar with the problem domain and the target technology. I needed to learn about some of the fundamentals of digital signal processing, the domain of diagnostic cardiology, and pacemakers themselves.
I started by examining the data that we received from a clinical investigation with our new investigational device. I quickly discovered that I needed to invent a filter that would remove the injected lead fail detection carrier signal, before I could view the ECG with the pacemaker pulses. Once I completed that task, I was able to review the data.
After reviewing the data, I reviewed the original prototype algorithm and ported it into my system, at a higher sampling rate. The original algorithm only worked on uniphasic pacemaker pulses, and it could not handle biphasic pacemaker pulses. I started working on adapting the algorithm that could handle biphasic pacemaker pulses.
A pacemaker pulse can be modeled as a Markov process, with a basic left to right model. I created a deterministic finite state machine to handle the different states of the Markov process, and I used fixed criteria for the transitions from state to state. Once I completed the uniphasic model, I adapter for the biphasic case. Then, I completed a series of experiments where I adjusted the initial design logic to match the training data set. Once, I had the algorithm working well over the training data set, I completed testing on the test data set. Further adjustments to the FSA were required, to account for specific noise conditions, to account for rare pacemaker pulse morphologies, and to account for performance issues.
The final algorithm detects pacemaker pulses from atrial, ventricular and biventricular sources, with the desired improvement in sensitivity and positive predictive accuracy for pacemaker pulse detection. The clinical validation of the algorithm showed an increased sensitivity when compared to the legacy system.
Like most of the projects that I start, I was unfamiliar with the problem domain and the target technology. I needed to learn about some of the fundamentals of digital signal processing, the domain of diagnostic cardiology, and pacemakers themselves.
I started by examining the data that we received from a clinical investigation with our new investigational device. I quickly discovered that I needed to invent a filter that would remove the injected lead fail detection carrier signal, before I could view the ECG with the pacemaker pulses. Once I completed that task, I was able to review the data.
After reviewing the data, I reviewed the original prototype algorithm and ported it into my system, at a higher sampling rate. The original algorithm only worked on uniphasic pacemaker pulses, and it could not handle biphasic pacemaker pulses. I started working on adapting the algorithm that could handle biphasic pacemaker pulses.
A pacemaker pulse can be modeled as a Markov process, with a basic left to right model. I created a deterministic finite state machine to handle the different states of the Markov process, and I used fixed criteria for the transitions from state to state. Once I completed the uniphasic model, I adapter for the biphasic case. Then, I completed a series of experiments where I adjusted the initial design logic to match the training data set. Once, I had the algorithm working well over the training data set, I completed testing on the test data set. Further adjustments to the FSA were required, to account for specific noise conditions, to account for rare pacemaker pulse morphologies, and to account for performance issues.
The final algorithm detects pacemaker pulses from atrial, ventricular and biventricular sources, with the desired improvement in sensitivity and positive predictive accuracy for pacemaker pulse detection. The clinical validation of the algorithm showed an increased sensitivity when compared to the legacy system.
Sunday, June 7, 2009
Life Stories #3 - Marquette University Master's Thesis
This is a very brief summary on the process of completing my thesis at Marquette University...
I decided to do the thesis option for my Masters of Science in Computing at Marquette University. I choose a thesis topic from a set of topics for the Dr. Dol0ittle project. This is a project to apply speech-processing techniques to animal vocalizations. I decided to tackle the problem of generating a system to automatically determine frame size, frame rate and the number of HMM states given a sample sound. After a preliminary meeting, my adviser and I decided that the best method would be to use instantaneous frequency and instantaneous bandwidth to estimate the frame size and frame overlap for a sample animal vocalization.
This is a research project, so I needed to learn about estimating instantaneous frequency and bandwidth, estimating frame length and frame overlap, and estimating the number of HMM states. I needed to do this with minimal assistance from by thesis adviser because I was working full-time and off campus most of the time.
First, I searched as much information about estimating frame length and overlap as I could find in the literature. Also, I searched and read multiple papers on methods for estimating instantaneous frequency and bandwidth, and for estimating frequency spectrum. I also read most of a book on instantaneous frequency estimation. Finally, I researched for articles on HMM state estimation.
Once I sufficiently researched the previous methods tried by other scholars in the field, I designed a plan for completing the project. I discussed the plan with my adviser and he liked the idea.
After I reviewed the idea with my adviser, I started the process of writing the code needed to implement the plan. In this case, I wanted a framework where I could perform the estimation on a list of animal vocalizations and review the results of the algorithm. I wrote a C++ program using the GCC, automake, autoconf and the GSL to perform the automatic frame length, frame overlap and HMM topology. Then, I wrote a Python script to run the various experiments, where I changed specific parameters of the algorithm to control the frame length, frame overlap and HMM topology on an algorithm by algorithm basis.
After running all of the experiments on the target animal vocalizations, I finished writing my thesis paper and presented it to the thesis advisory board. The results showed that the frame length, frame overlap and HMM topology could be estimated using a single animal vocalization, and that this technique could be generalized across species.
In the end, the thesis advisory board liked my presentation and paper, and I completed my thesis and graduated.
I decided to do the thesis option for my Masters of Science in Computing at Marquette University. I choose a thesis topic from a set of topics for the Dr. Dol0ittle project. This is a project to apply speech-processing techniques to animal vocalizations. I decided to tackle the problem of generating a system to automatically determine frame size, frame rate and the number of HMM states given a sample sound. After a preliminary meeting, my adviser and I decided that the best method would be to use instantaneous frequency and instantaneous bandwidth to estimate the frame size and frame overlap for a sample animal vocalization.
This is a research project, so I needed to learn about estimating instantaneous frequency and bandwidth, estimating frame length and frame overlap, and estimating the number of HMM states. I needed to do this with minimal assistance from by thesis adviser because I was working full-time and off campus most of the time.
First, I searched as much information about estimating frame length and overlap as I could find in the literature. Also, I searched and read multiple papers on methods for estimating instantaneous frequency and bandwidth, and for estimating frequency spectrum. I also read most of a book on instantaneous frequency estimation. Finally, I researched for articles on HMM state estimation.
Once I sufficiently researched the previous methods tried by other scholars in the field, I designed a plan for completing the project. I discussed the plan with my adviser and he liked the idea.
After I reviewed the idea with my adviser, I started the process of writing the code needed to implement the plan. In this case, I wanted a framework where I could perform the estimation on a list of animal vocalizations and review the results of the algorithm. I wrote a C++ program using the GCC, automake, autoconf and the GSL to perform the automatic frame length, frame overlap and HMM topology. Then, I wrote a Python script to run the various experiments, where I changed specific parameters of the algorithm to control the frame length, frame overlap and HMM topology on an algorithm by algorithm basis.
After running all of the experiments on the target animal vocalizations, I finished writing my thesis paper and presented it to the thesis advisory board. The results showed that the frame length, frame overlap and HMM topology could be estimated using a single animal vocalization, and that this technique could be generalized across species.
In the end, the thesis advisory board liked my presentation and paper, and I completed my thesis and graduated.
Tuesday, June 2, 2009
Life Stories #2 - Learning to Program
Back when I was in junior high school, I decided that I wanted to learn how to program computers. Our high school just received a couple of TSR-80 computers, and I wanted to get into the “computer lab” to learn how to program computers, and to play some computer games that were rumored to be in the lab.
At that time, I had never programmed a computer or worked with one, so I was starting from scratch. I went to the library and picked-up a book on programming in BASIC. I read the book, and soon I was writing small programs in BASIC on the TSR-80, and saving them on the large 5” floppy disks. I made at attempt at writing a basic maze game where you navigate through a maze using only a text interface.
Later on, I saved enough money to purchase my own VIC-20 personal computer. Again, I learned how to program in assembly and wrote a very basic, millipede-like game, using pixel graphics on the VIC-20. By the time I was enrolling in a programming class in high school, I was able to skip the level 1 programming class and move onto the level 2 programming class, without taking any type of test. My math teacher knew my capabilities and wouldn't allow me into the level 1 class. In the level 2 class, I wrote a program that generated random numbers as a substitute for dice for role playing games.
The result of all of this self-initiated learning is a career in Software Engineering, where I continue to learn new technologies and languages and apply that learning to solve problems. I have been doing this for 17 years, and loving every minute of it!
At that time, I had never programmed a computer or worked with one, so I was starting from scratch. I went to the library and picked-up a book on programming in BASIC. I read the book, and soon I was writing small programs in BASIC on the TSR-80, and saving them on the large 5” floppy disks. I made at attempt at writing a basic maze game where you navigate through a maze using only a text interface.
Later on, I saved enough money to purchase my own VIC-20 personal computer. Again, I learned how to program in assembly and wrote a very basic, millipede-like game, using pixel graphics on the VIC-20. By the time I was enrolling in a programming class in high school, I was able to skip the level 1 programming class and move onto the level 2 programming class, without taking any type of test. My math teacher knew my capabilities and wouldn't allow me into the level 1 class. In the level 2 class, I wrote a program that generated random numbers as a substitute for dice for role playing games.
The result of all of this self-initiated learning is a career in Software Engineering, where I continue to learn new technologies and languages and apply that learning to solve problems. I have been doing this for 17 years, and loving every minute of it!
Wednesday, May 20, 2009
Life Stories #1 - Designing a Decimation Filter with MATLAB, without the Filter Design Toolbox
I am working through the activities in the book "What Color is Your Parachute?" by Richard Nelson Bolles. Part of this effort is to write seven stories about a goal that I wanted to accomplish, and what I did to accomplish it. I decided to start with a recent story. It is titled:
Designing a Decimation Filter with MATLAB without the Filter Design Toolbox
About a year ago, I had a goal to create a better decimation filter for a particular product. This product had an over-sampling system, where eight electrical signals were sampled at a rate 8x higher than needed, and decimated to the desired target rate.
At the time, I understood the basics of FIR filter design, but I never actually designed or implemented an FIR filter. Being a lifetime-learner, and a having a “no fear” approach to software, I attempted to design the filter using only my basic knowledge of FIR filtering, a copy of “Understanding Digital Signal Processing” by Richard Lyon, occasional advice from a seasoned DSP pro, and an abundance of “stick-to-itiveness”.
In summary, I designed the filter as a two stage FIR decimation filter. The first stage reduced the sampling rate by a factor of 4x and the second stage reduced the sampling rate by a factor of 2x. A single FIR filter would have required 96 taps to properly band-limit the signal for decimation. A two-stage FIR filter reduced the required filter taps to only 66 taps total.
I created the filter by modeling the frequency response of the front-end analog filter with a Butterworth filter design in MATLAB. Then, I inverted the frequency response over the bandwidth of interest, and I equalized the frequency response to reduce the attenuation of the hardware filter over the target bandwidth.
I added the frequency response of the equalization to the frequency response of the first stage of the FIR filter, and I used the “fir2” function with a Chebyshev window to design the first stage of the FIR filter. I experimented with the number of coefficients until I had the desired frequency response for the first stage of the decimation filter.
Next, I used the same process to generate the coefficients for the second stage of the filter (i.e., determine the combined frequency response of hardware filter and the first stage analog filter, inverse this response over the bandwidth of interest, adjust the desired frequency response for the proper stop band, and use the “fir2” function to generate the filter coefficients).
Finally, I implemented the filter function in C and added the two stages of filtering to the product. During the source code implementation, I created a set of unit tests, using CUTEST, that verify the functionality of the two-stage decimation system.
The result is a decimation system that equalizes the target passband. Analysis showed that the new decimation system is much better that the old-boxcar approach. The filter, in combination with the analog front-end filter and the system gain, provides full gain over the target passband and –100 dB of attenuation at the Nyquist frequency.
A series of frequency response tests provided that the two-stage decimation filter reduced the aliasing significantly in both the real-world system testing and unit testing. It accomplished this without reducing the target bandwidth for the system.
Designing a Decimation Filter with MATLAB without the Filter Design Toolbox
About a year ago, I had a goal to create a better decimation filter for a particular product. This product had an over-sampling system, where eight electrical signals were sampled at a rate 8x higher than needed, and decimated to the desired target rate.
At the time, I understood the basics of FIR filter design, but I never actually designed or implemented an FIR filter. Being a lifetime-learner, and a having a “no fear” approach to software, I attempted to design the filter using only my basic knowledge of FIR filtering, a copy of “Understanding Digital Signal Processing” by Richard Lyon, occasional advice from a seasoned DSP pro, and an abundance of “stick-to-itiveness”.
In summary, I designed the filter as a two stage FIR decimation filter. The first stage reduced the sampling rate by a factor of 4x and the second stage reduced the sampling rate by a factor of 2x. A single FIR filter would have required 96 taps to properly band-limit the signal for decimation. A two-stage FIR filter reduced the required filter taps to only 66 taps total.
I created the filter by modeling the frequency response of the front-end analog filter with a Butterworth filter design in MATLAB. Then, I inverted the frequency response over the bandwidth of interest, and I equalized the frequency response to reduce the attenuation of the hardware filter over the target bandwidth.
I added the frequency response of the equalization to the frequency response of the first stage of the FIR filter, and I used the “fir2” function with a Chebyshev window to design the first stage of the FIR filter. I experimented with the number of coefficients until I had the desired frequency response for the first stage of the decimation filter.
Next, I used the same process to generate the coefficients for the second stage of the filter (i.e., determine the combined frequency response of hardware filter and the first stage analog filter, inverse this response over the bandwidth of interest, adjust the desired frequency response for the proper stop band, and use the “fir2” function to generate the filter coefficients).
Finally, I implemented the filter function in C and added the two stages of filtering to the product. During the source code implementation, I created a set of unit tests, using CUTEST, that verify the functionality of the two-stage decimation system.
The result is a decimation system that equalizes the target passband. Analysis showed that the new decimation system is much better that the old-boxcar approach. The filter, in combination with the analog front-end filter and the system gain, provides full gain over the target passband and –100 dB of attenuation at the Nyquist frequency.
A series of frequency response tests provided that the two-stage decimation filter reduced the aliasing significantly in both the real-world system testing and unit testing. It accomplished this without reducing the target bandwidth for the system.
Subscribe to:
Posts (Atom)
