Posts - Learn It, Make It

Latest Posts

The FreeRTOS Tick – Unravel the Mystery to Master the Tick

Published by Joe

Over the last few years, I have become a fan of FreeRTOS for a few different reasons:

It’s free (though a paid versions allow for some protection)
Portable as it abstracts well from the hardware it runs on
Provides an easy way to separate sections of code
Allows for control of when, and how often, tasks are run

It’s the last point that this post will delve into. How does each thread (tasks in FreeRTOS speak) control when they run? It all comes down to the ‘tick,’ the magical trigger that FreeRTOS uses to know when to switch which task is running or give up waiting for a semaphore that will never be given.

What Drives the FreeRTOS Tick

It all comes from the FreeRTOS Tick – the function that is called at a periodic interval. As we will see below, this tick drives the context switch between tasks. But first, what drives this tick?

At the simplest level, the tick is just a timer that has a period set to match the desired tick rate. By default, this is set to 1ms which provides a good balance between task speed and overhead of task switching. The timer interrupt is triggered every time the period is hit. The ISR calls into FreeRTOS’ vTaskSwitchContext() function which we dive into below.

For ARM core processors, there is a special timer designed specifically for the purpose of providing an RTOS its tick. This timer is called the SysTick (System Tick) and provides its own clock configuration, counter value, and interrupt flag. This allows you to set up the SysTick to provide FreeRTOS’ tick and use all the other timers for your program’s needs.

But as we will look at in future posts, we can utilize the flexible nature of what drives the FreeRTOS tick to reduce power.

What Happens at the FreeRTOS Tick

So coming back to the vTaskSwitchContext() function that is called during the timer tick, what happens when the tick happens?

While we won’t focus on the nature of cooperative and preemptive multitasking. You can check out the links provided to get a review of these multitasking models. FreeRTOS allows for both models, but our discussion on the tick makes more sense in the preemptive mode.

In FreeRTOS, you create ‘tasks’ that contain a functional component that you want to run at specific times or in reaction to other events. When the operation system is running, each of these tasks exists in one of four states:

Running – the process currently doing work
Ready – is available to run as soon as the OS allows it
Blocked – waiting for an event to allow it to run
Suspended – not even checked by the OS, task is shut off

When the tick triggers vTaskSwitchContext() the kernel looks at which tasks are in the Ready state and if there are any higher priority tasks than the current one, it switches context to allow the task time to run.

The diagram below shows a basic example of three tasks switching context either due to priority interrupt or context switching during the tick.

Key: P = Preemption, S = Switch, * = task has gone into blocked state

Higher priority tasks (higher number in FreeRTOS) are immediately switched to if they become unblocked due to a semaphore or queue releasing them and the next task takes over if any task goes into the blocked state (waiting for a semaphore or queue).

Tick Drives Round Robin

The FreeRTOS tick also looks for any other tasks that are equal in priority and will give each a chance to run. If multiple tasks are the same priority and are always in the Running or Ready state, the kernel creates a Round Robin model where each gets a full tick before switching to the next.

This Round Robin mode is where the balance between how fast the System Tick should be. A faster tick would allow more tasks to do operations in a given timeframe, but each task will only be able to do a limited amount of work before having to stop. And while the kernel is small and optimized for microcontrollers, it still takes a certain amount of time for it to save the current task’s context and load in the next task.

Ticks can Unblock Tasks

While the FreeRTOS tick function generally is switching context between equal priority tasks, it could also cause tasks to go from the Blocked to the Ready state, potentially moving a higher priority task into the Running state. One way this happens is if a task has called vTaskDelay() which delays a certain number of ticks.

Each call to vTaskSwitchContext() checks if any task is blocked due to the delay function and update’s its state if it has elapsed the desired time.

What if Nothing Needs to Happen

For many systems, especially low power systems, you don’t want tasks constantly running. In part, this happens because tasks are generally setup to react to certain events, events that may not happen often.

Between waiting for the next event, it could be that all of your tasks have nothing to do and have entered the Blocked state.

The Idle Task of Last Resort

When this occurs, FreeRTOS’ idle task runs instead, which is where FreeRTOS does some garbage collection for systems that create and delete tasks during runtime. Otherwise, the idle task is where you do low priority tasks like diagnostics or logging. This code could be called by the idleTaskHook function inside the Idle task, or creating tasks that are set to the idle priority of 0.

Looking at Lower Power

What we will get into later is looking at how long we need to wait for the next task to be serviced or times out waiting. We can use this information to decide to go into lower power states to conserve battery life or improve power efficiency.

At the simplest level, the ARM call to WFI will put the processor on hold until the next interrupt kicks it back into gear. This saves a small amount of current as the core processor clock is disabled until the next FreRTOS tick.

Shortly we will have a post focused strictly on the low power aspect of FreeRTOS. If you want to make sure you notified of these upcoming posts, sign up for our weekly newsletter below.

Full-Scale Production – New Product Introduction in Seven Steps

Published by Joe

This is the final step in our series on New Product Introduction: Full-Scale Production. If you missed the rest of the series, get caught up in the intro post.

So here we are, after months of hard work, trips to the manufacturer, and probably long nights, we are finally at the last stage of New Product Introduction: Full-Scale Production. While it could be argued that at this stage you are no longer in NPI mode, I view the start of scale production as the transition zone.

Full-Scale Production

It is now time to turn the crank and print the money. This is where the manufacturer is building hundreds or thousands a day to be shipped out to customers. The processes have been worked out, the hardware and firmware bugs have been solved or accepted, and now it’s just copy and paste.

Go ahead and pop the champagne and celebrate, this is an exciting milestone, one that many products never achieve.

But Now What?

But, the work isn’t over here – though many an engineer wished it was. The product has merely crossed the NPI line and moved into the sustaining phase of a product’s life.

Sustaining Engineering

Almost a dirty word – well at least to me, others find this part of the product life cycle fascinating, so I will do my best to do it justice.

If you are at a larger company, the sustaining engineering is usually taken over by a dedicated manufacturing and test team. A team of these specialists that love to tweak the process and adjust the BoM cost so that every little penny and production second can be eeked out of the product.

On a smaller team or startup? Well then take notes because you get to put on the operations hat.

Cost Optimization

The first role of the operations team (or your new hat) is to continue to cost optimize the product. During development, there were choices made that allowed for expedient delivery to manufacturing but may not have taken cost into consideration.

This isn’t about changing out a cheaper processor (which involves a good deal of work) but rather finding compatible parts that can be either swapped in or allow the chip vendors to fight over providing you with the lowest cost.

A few cents saved doesn’t seem like a lot until you multiply it by the thousands or millions you may be building. And a few pennies here, a dime or two there, and eventually it adds up to dollars saved.

Another route to cost cutting is improving the production process flow. Pre-programming the chips on specialized reel-to-reel machines speed up test time. Building additional automated test fixtures allow for more throughput. All these things reduce the manufacturer’s overhead, and in turn, your product’s cost.

BoM Management

Speaking of trying to find crossmatched components, the operations team is also responsible for dealing with the end of life or extremely long lead time parts.

We talked about this a bit in one of the early stages of NPI in the Part Availability step. But it’s not easy to predict when or why parts are obsoleted or suddenly become in short supply. So at times, they must be dealt with in manufacturing.

For simple parts like passives, a crossmatch should be easy to find. Other times it is a more complicated part such as an IC. In these instances, a small redesign of the board may be required switches in a new part that does the job but doesn’t fit right into place.

When the redesign is required, a smaller version of the NPI process is repeated that allows for the change to be tested and verified to fix the problem. This process usually doesn’t take nearly as long or as complicated – which is good because it is fixing a short-term problem of part availability.

Failure Analysis

Even with all the development testing and manufacturing test that is done on the product, failures will continue to happen. Sometimes this is just a yield issue on the parts you are using.

As an example, one of the products I was involved with had a small percentage of the products stop working in the field. Every failure came back to a capacitor blowing, a part that had plenty of margin in the voltage range.

It came down to the capacitor manufacturer had redesigned their formula without changing the part number and were seeing a small number of components released that were well above the tolerance level.

It’s not until you are building the thousands and millions that these problems start to appear. The capacitor vendor only saw a 0.1% failure rate, but as we were using a number of these capacitors in our product, we saw closer to 1% field failure

What Happens Next?

OK, so we talked about what the operations role would be doing next, but what about all the developers? What do they do when a product roles into full production?

Well, for some development engineers, they would move on to the next product. A strong hardware product company will always have products in the pipeline. So as one exists into production, a new one is started.

At other times, the operations team may tap development team for changes, end of life parts, or systematic failures. This may be the case if the PCB needs to be tweaked for a replacement part or the firmware locks up three months down the road. It’s a bit of sustaining engineering and product development mixed.

End of the Road

For the product, it will stay in Full-Scale Production for a long while, continuing to churn out of the factory as long as the customers keep buying and your company continues to support it. Eventually, though, every product reaches end-of-life. Perhaps because a new generation is being released or it could be not enough people bought it to justify the support required.

But no matter the reason, the End-of-Life stage is the concluding, eventual end of the Product Life Cycle – the larger process a product goes through (NPI just being an early stage).

Bringing it All Together

So it took us seven steps to get here, but together we have gone through what is needed to bring a prototype into full production – well at least from the engineering side (I’ll leave the legal and business side to another time).

We went from the prototype that was checked for manufacturability, to building a few runs that verified everything worked, to figuring how to test the product before it went out the doors in full-scale production mode.

New Product Introduction isn’t an easy process for a company to go through, but I hope that by breaking down each stage I was able to help you walk through what was important at each step. As with any engineering challenge, by breaking it down each subcomponent is easier to understand and accomplish – building to a greater whole.

Product Verification Build – NPI in Seven Steps

Published by Joe

Product Verification is part six of our New Product Introduction series. If you missed the previous posts and want to get caught up, start with the introduction post.

At this stage of the New Product Introduction process, your manufacturer has built small, hands-on runs of your product. They have put their best team on the lines, and probably had a high level of oversight at every step.

This has made for very expensive units, probably twice as much as you were expecting. But as described in the Design Verification stage, it was all investment in making sure that everything would run smoothly once the crank started turning.

Product Verification Build

And that cranks is beginning to wind up. The Product Verification build (or just PV) is much like a mini-production run. In fact, the PV is a practice run before releasing the production flood.

This means that what you receive from this build is a product that could, in theory, be placed on a store shelf. As we dive in, we will find that while you could sell these units, they will provide valuable to your manufacturer and your company.

What Your Manufacturing Is Doing

When full production starts, the manufacturer wants to be able to throw anybody trained with their procedures on the line and potentially crank out your product 24/7. To do this, they will be pulling back their A-Team and allow the shift line workers to run through the full process themselves.

Switching in the line workers makes sure all the process documentation is entirely clear, and any delays in the procedure are resolved. By bringing in new eyes to the process, they can ensure any assumptions are cleared up with documentation and every step is clear and efficient.

But this run is still limited to a specific number of units – perhaps a few thousand. The number is usually set by how many would build in a single batch or fill a pallet. Again, this provides practices for both the manufacturer’s shipment department, but also your logistics group as it prepares to handle the new inventory.

What to do with the PV Units

Engineering and operations will take a random sampling to ensure everything was built correctly, marketing and sales will take some to give to insiders, and others will be handed out to employees, friends, and family to check out before releasing to the public.

It is perhaps this friends and family group that will provide the most benefit to the company as they may be the first customer-like people to interact with the product. Having this larger group of people interact with the product and maybe with the larger system as a whole your team could work out more bugs and features that could be fixed before full production runs.

Giving the OK

Until your team gives the OK, the manufacturer will be waiting. They will review how the process went on the line and true-up any difference in costs, but they will mostly be waiting to charge ahead.

This is a bit of a scary point as an engineer. Until now, the product lived in the lab or given to ‘safe’ people. This is the last chance to make any changes before it becomes ‘real.’

But unless there is something that absolutely must be modified, nothing should be touched. Not only are changes going to be extremely expensive at this point, but everyone involved (and it’s a lot of people at this point) are anxious to move forward – and causing a step backward will not be looked at favorably.

So if you want to tweak something, just a bit more, forget it and accept that this version of the product won’t be perfect.

Product Verification Result

At the end of the PV build your team will have hundreds or even thousands of product ready devices to hand out, test, or give back to the operations team to sell.

Your manufacturer will have worked out final procedures and costing for each device.

We could even consider this point the end of the New Product Introduction process as the only thing left is to move into full production, but as we will see in the next post, there is still some work to complete before printing the money.

Customer Service for Engineers

Published by Joe

Customer service often gets a bad rep in the press. Unfortunately, it’s often due to insanely poor service from corporations, companies that win Consumerist’s Golden Poo Award. It could be in part because these large businesses outsource the role to call centers that get paid by volume of handled cases rather than the value provided to the customer.

But that isn’t the service I want to talk about today. Instead, I want to help engineers understand what the importance of good customer service and how people in the role promote your company’s brand.

On the Front Line

A dedicated team may hold the role of customer service or shared by other roles within the business. Customer requests may come in through emails, support tickets from a web form, calls, or even passionate customers showing up at the front desk.

Whoever is filling the role, they are the front line of your company with its customers. Consumers have become to expect a high level of product quality and when something doesn’t work right out of the box, passion flares and angry emails to the company are sent.

I’ve never been in war, but I can imagine it to be a very scary place, full of emotion and false steps leading to a quick end. And perhaps putting customer service out on this front line is a bit superfluous, but when they are getting angry emails and angry calls, I’m sure they fell as they are on the front.

And one wrong move from them (as seen so often in the media) can create torrents of bad PR for the company. On the other hand, a skilled customer service professional can turn the tide and even create new fans of your company – all with a few well-placed responses.

Triage Issues and Escalate

Functionally, the customer service triages issues and attempts to provide answers to customers. Sometimes this is classed into ‘tiers’ where each layer responds the best it can and escalates what it can’t.

Call Centers, FAQ
Support specialists
Developers

Call centers, FAQ pages, and automated responses provide tier 1 support and could solve the majority of problems customers face. Companies that care about their customers will then provide an easy way to connect the customer support team through the phone or email.

This team is providing tier 2 support, helping customers diagnose problems, dive into support dashboards, and use their experience to provide help. A well-equipped support team could offer guidance and solve the majority of issues faced by customers.

But sometimes funny shit happens like a new build went out without adequate testing. Products that typically work great, suddenly stop. Customer support is at a loss. At this point, the problem is escalated to tier 3 which often involves developers or, at least, their managers. This level can dive into the code and see where bugs popped up or provide technical reasons why something is failing.

The balance that companies face is the balance between triaging as much as possible to minimize developer distraction, and providing enough transparency that developers see what customers are dealing with. An excellent customer service team can provide customers with answers and be their advocate to the engineering team.

Provide a Voice of the Company

People in sales or marketing may get a bit upset about this statement as being ‘the voice’ is part of their job description. But while sales and marketing bring new customers in, it is customer service that makes or breaks their long term commitment.

Advertisements and other marketing channels bombard customers all the time with messages to buy this product or use that service. But these don’t provide any human relatability or engender the trust that fans require. Put them on the phone with a good customer service rep, on the other hand, and the get to hear the company.

Give them a High Five

So next time you see someone from your customer service team in the hallway, give them a high five for representing your company. And when they come to you for technical assistance, provide the best help that you can, because you are helping them win a customer over – a customer who could be an influential enough person to increase drastically sales.

NPI in Seven Steps – Design Verification Run

Published by Joe

Design Verification is part five of our New Product Introduction series. If you missed the previous posts and want to get caught up, start with the introduction post.

Your team has now received the Engineering Verification units and your manufacturer has completed their test plan to ensure they are building the product correctly. You’re now ready to build millions, right?

Now quite. You have a nice prototype that may even look like a product, but you and your manufacturer need to go through a few more steps before reaching full production, well at least if you want to get it right.

At this stage of the New Product Introduction, we are going to build what is called the Design Verification build, or just DV. After completing the EV run, your team and your manufacturer probably had a few design changes needed to get the product working right and able to test efficiently. If those changes were significant, you might have had to redo the EV build, but otherwise, the changes get implemented and the DV build gets kicked off.

This will still be a relatively small run, maybe twice the size of the EV, but still less than a hundred units or so. The difference is that the design will be as close to the finished product as possible. This includes enclosures, PCBs, and any other user interfaces. But not necessarily labels and packaging.

The point of the DV run is to ensure that any changes made during the EV build were implemented correctly and that the ICT, optical, and/or the FCT are ready to go. As we’ll see below, there are a few additional tests that you are able to complete with these units.

Run the Process

For the manufacturer, the Design Verification build is a very hands-on run. They are spending as much time documenting and rehearsing each step of the process as they are building the product. They do this as an investment in making sure that everything will go smoothly when they start full production.

The build process that the manufacturer is creating is like a recipe that they will use when they start to crank out units in the thousands. The better they can refine the recipe now, the smoother it will be when they start production as it’s easier to work out the bugs now.

Refine the Design

For your team, this build will allow any refinements in the design to be tested and the marketing team can see what the final package will look like (the pretties are very important to them). This is a step past the EV units as it would include the first tooled enclosures (may only use ‘soft tooling‘) and the size and features meet what the end product will be.

As I talked about in the Fail Fast post, seeing and holding a product make it real to people so this build provides that ‘real feel’ experience. This is important as minor bumps in the plastic can be felt but not seen in a CAD program. Seeing how bright the LEDs are or how loud the speaker is may drive additional changes now that it is in the plastic housing.

Certification Testing

For many certifications, you are required to provide a product sample that is in the final housing. As the DV build is the first build that would include this feature, it is often the build that is used for any certifications your device needs.

For any product sold in the US, this would include FCC or the CE mark for the EU. These certificates can take weeks to complete the test, write the reports, and wait for the governing body to provide the stamp of approval. This long delay is why the DV (rather than later builds) are used.

Testing may also be done that isn’t necessarily a certification requirement but more for product durability. This could include:

Drop and vibration testing (make sure nothing breaks off)
HALT/HAS (attempt to accelerate component life)
Environmental (does it mold or melt)
ESD tolerance (does it survive a simple static shock)

Some of these tests you may have started with the EV build, but may need to get repeated with the new design (and extra units that have arrived).

Could you Just Skip the Design Verification?

Let’s say that your EV run was perfect, no changes were needed and the manufacturer has a fully working FCT in place. Do you need a Design Verification run?

The answer, like so many in life, is that it depends on your level of risk tolerance. The DV build gives your team and the manufacturer another low-unit run to make sure everything is ready to go before making huge commitments. Each build will become larger (and thus more costly) and if bugs are found in the hardware or pre-installed firmware, it’s that many more that will need re-work or be scrapped.

From: Sustainable Product Innovation: The Importance of the Front- End Stage in the Innovation Process

As the above graph shows, the further you go, the higher the cost of creating changes will be. These costs are both monetary (you might need to change fixtures, change plastics, or buy new parts) but also in time to build new items.

So if the team is in a rush to get to production and the EV build was a success, then the risk vs. reward of skipping the DV may be low. On the other hand, if money is tight or new features are being added or changed, the DV may be the best option to wring them out.

Design Verification End Result

At the end of the Design Verification build, your team will have on hand a large number of almost-product-ready units. Units that can be shared more broadly within the company as it is starting to look ‘nice,’ as well as with key business partnerships.

It’s not just about the pretty factor, the DV build also helps your manufacturer get ready for production. When they have completed the build they should be able to provide a more accurate price quote for their overhead and output ability. Usually, this trends in the positive direction for your company as most manufacturers tend to buffer the estimate, but if it turns out your product is more complicated than originally thought (or feature creep) they may work with you to adjust the design or accept a higher cost.

With a successful DV run behind you, both your team and the manufacturer can start to gear up for product release and full-scale production. But standing between you is one more build: the Product Verification build which we will talk about next.

Fail Fast, Succeed Fast

Published by Joe

The micro-world that is Silicon Valley has a perception of a breeding ground for new ideas. With a few horrendously bad (like ‘disruption‘), and others being quite good – when put into context.

Fail fast! Fail often!

This is an idea that has been bandied recently, and out of context, some companies have used it to fail right out of business.

The core idea of this concept is to make small, but brave changes to your project and see how the customer responds. Only a small amount of time is invested and if the result is an awful failure there has been little lost.

On the other hand, if the results look promising, you could iterate the idea into a successful product. Fail fast – but not for the sake of failing. Fail fast or succeed fast, either way, the idea is to take an idea and act on it.

Fail Fast in Your Career

For you, the idea of fail fast is just as applicable to your career. However, for performance review reasons, we may want to rephrase it as “pushing growth.” By this, I mean that to grow within your own skill set and push any project forward, you need to take risks – risks that something may fail.

Similar in nature to a previous post on Making Mistakes, taking the risk of something failing is often the only way to move forward.

Think of a legacy section of code or process that your company uses. Something that is so crusty (and yet works) that no one wants to touch it with an 83 ½-foot pole. Yet, that code continues to be a weak spot in the product and workflow – like a mild migraine that doesn’t go away.

What if you took a chance to slice off a small part of the crust and redesigning a feature to make it more robust? It may fail horribly, but you could also make big contributions.

Destroy the Endless Meeting Drone

Another way to think about using the fail fast approach is when meetings drone on about whether a feature could work (like would blue be better than green). I’ve been in too many meetings where the same slides are presented on the pros and cons of a feature and the same arguments are rehashed.

Nothing ever gets done. But, throw something together that approximates the features and present it at the next meeting? Even if it doesn’t work, your team may be impressed with your initiative (and thankful the meeting can finally end).

Quick and dirty demos can win out over well-prepared slides any day. Just be prepared for the reactions to range from ‘that idea is awful’ to ‘we are geniuses!’

Seeing is Believing

It comes down to the fact that seeing and interacting with something is more ‘real’ than talking about it. The key is that it doesn’t have to be complete, spend just enough time to approximate the idea and allow for quick feedback.

Henry Ford was famously quoted as saying “if I asked what people wanted, they would have said faster horses.” It’s easy to think back on that and think the first Ford car was this perfect marvel of engineering, so well thought out that it completely transformed everything in life.

The reality is that Ford threw together existing ideas (there were many other companies building automobiles) and quickly churned out something new – the assembly line. When the first Model A was sold, Ford only had $223.65 left in the bank (ref) – he had to get production done quickly and start selling. The Model A had multiple limitations (some could say failures), but they sold and they could build more (successes).

Fail fast, but also succeed fast

The whole point here, and why this mantra has resonated with some successful businesses, is that having something, anything to show can be better than hand waving and slides. So take some initiative and build that thing that never goes away in meetings.

NPI in Seven Steps – Production Test, FCT and ICT

Published by Joe

Production Test Verification is part four of our look at New Product Introduction, if you missed the previous articles, get caught up here. In this post we use the prototypes from the Engineering Verification build to start the production test procedures and fixtures.

At this point in our process through the New Product Introduction, we just received our first manufactured boards from the Engineering Verification run and are giddy with excitement I’m sure (or frustrated with hardware bugs!).

Your manufacturer is also excited about the boards as well so they can start the next step in the NPI process: designing and producing the Production Verification Test.

What is Production Test Verification

As we previously hinted at in the Design for Manufacturing and Test phase, the manufacturer has an incentive to make sure every product that goes out their door works to the best of your design. This is where the Production Test Verification comes in.

While the EV build isn’t the final product, it still gives the manufacturer something to work with to figure out how and what to test. This will help drive changes in the next build that would enable more thorough testing.

Production Test != Unit Test/Design Test

A quick note of what Production Test Verification is not. When a manufacturer is looking at testing the product, they aren’t talking about unit tests or design verification. This is what your team does with the EV units and during normal development. You hammer on the edge cases, the raw performance metrics, and error handling.

The manufacturer simply wants to test to make sure the board was built correctly and any firmware was installed. They are looking for as simple of a test that can verify they didn’t screw anything up in assembly or need to adjust parameters in their system.

Types of Production Test Verification

To accomplish this simple verification test, the manufacturer has a few different types of tests to call on. The three most common are listed below, in order of increasing complexity and cost.

Visual Inspection (optical/x-ray)
In-Circuit Test (ICT)
Functional Test (FCT)

Each has its strengths and weaknesses so part of this phase of the NPI process is selecting the which tests will provide the best test coverage balanced with the cost of test development and per-unit test cost. We go in depth on each below.

Visual Inspections

The simplest and most straightforward method of making sure the product was built correctly, is to simply look at it. This could be as crude as having technicians manually inspecting each piece to impressive pieces of equipment that automate the process.

Optical verification uses cameras to look at each component on a PCB carefully and compares it to a pre-programed image of how it should look. Each image can verify the component was placed in the correct orientation, check the part markings to make sure it is the right one and even look at the solder connections to verify they are correct.

Production Test Verification by Visual Inspection using XRAY on BGA

By Martin Richard [Public domain], via Wikimedia Commons

X-ray is used on BGA parts since the contacts are located underneath the part. The x-ray looks for solder bridges (shorts between pins) and good contact between the pins and the pads – the two most common failures for this compact parts.

Benefits of Visual Inspections

Visual inspection is typically easy to set up as the machine reads in the placement files from the pick-and-place machines and records known good boards. With these fundamental data points, the machine is ready to go. By far the simplest of setups we will discuss here.

For parts that include part markings, this inspection can be an excellent first check to ensure that parts were placed correctly (correct value positioned in the exact spot and the right orientation). This could solve many ‘weird’ issues that occur in later tests as missing capacitors or resistors would fail the visual inspection.

Downsides of Visual Inspections

The downside of the optical and X-ray verification is time. Both take significant time to look over all the parts on the board, make an image comparison, and move the camera to the next location. X-rays also are also dangerous, so manufacturers require extra care whenever they are being used. Both time and complexity add cost to the production price, and thus your product.

They also only work well for parts that have value markings. Small resistors and capacitors typically don’t have any visual marking so the visual inspection can only verify it is placed, not if the wrong part was placed.

In-Circuit Test (ICT)

The next step up in complexity and evaluation is the In-Circuit Test (ICT). Similar to the visual inspection station, this is a specialized machine. Think of it like a big, complicated multi-meter that can quickly switch between hundreds of pins and check voltage levels, resistances, capacitances, and even inductance.

Your PCB would have a test point exposed on the board for every net that you want to test. The manufacturer creates a bed-of-nails that attach to their ICT machine which has spring-loaded pins. Each pin matches up with the test points on your board. These points could be via, larger pads, or traces with specific areas of solder mask removed.

By Miffo7uk [Public domain], via Wikimedia Commons

The machine would then be programmed to insert voltage levels at various test points within the design and measure the output at others. The voltage could be DC signals to measure resistance and shorts, or AC signals to measure capacitance and inductance.

Benefits of ICT

The benefit of the ICT is that it can verify most of the components were placed correctly and were the right values. This covers one of the weaknesses of the visual inspection in that exact values of components are measured.

In some ways, the ICT can replace the visual inspection as it can detect missing components or shorted nets. These errors would get detected by invalid measurement values at various nets which would allow the system to narrow down the problem and determine the error.

Downsides of ICT

The ICT has three major downsides:

Expensive to design the bed of nails
Significant amount of time to program the correct sequence of tests into the system
Requires a test point for every net exposed on the board which could drive the size and complexity of the layout up

Functional Test (FCT)

The highest level of complexity and customization is the Functional Test (FCT – yes the C is a bit odd).

The FCT runs the product through its paces, perhaps not as a customer would use it, but rather in a way that all the functionality can be verified. You or the manufacturer would build up a test fixture that would include pogo-pins (spring loaded pins), mounting posts, and some sort of clamp. The board can then be powered up, voltage and current levels checked, and digital communication piped into a computer.

For embedded products, this could include a particular version of firmware, or firmware running in a special test mode. This is the level at which the FCT is differentiated from the ICT in that the board would be active and tests the functionality rather than the raw values of the passives.

Benefits of FCT

The benefit of the FCT is that it tests the product much like your customer would, including testing the firmware, buttons, lights, and communication systems work correctly.

Where the ICT and optical inspection ensure parts are installed correctly, the FCT ensures that they are functional as well. While the ICT checks passive (RCL) values, it cannot verify that silicon components are functional.

The FCT is also a great place to program in serial numbers, calibration offsets, or other device specific values that need to be added to the specific product.

Downsides of FCT

The downside is that special test firmware will be required as well as the test infrastructure. The FCT also takes some time to run, adding additional cost for every facet that you verify.

The test infrastructure may require your company to buy test equipment for each testing station at the manufacturer and maintain your own computers to run the test software.

The End Result

In the end, your team and the manufacturer will decide on the best approach to testing the product which might include a combination of all the steps above – optical inspection of critical components and FCT of the user interface. And while every test adds cost to the product, shipping broken units because a part manufacturer sent the wrong reel, will cost even more.

The FAE, Sales Rep, Distributor and You

Published by Joe

As the New Product Introduction process starts, a confusing relationship starts to unravel between your company, your manufacturer and the maker of the parts you are attempting to buy. Somewhere in that mix, a distributor steps in to provide the needed parts to your manufacturer, sales representatives from all sorts of companies start calling on you, and your technical questions are redirected to mysterious FAEs.

But who are all these companies and how do they interrelate? And before you shrug it off and don’t think any of them matter to you (they’re ‘business people stuff’) I can tell you from experience that FAEs can be the difference between getting a part to work and hitting my head against a wall for weeks. And a good sales rep and distributor will not only help you find the right parts to meet your needs, they also drive pricing. Meaning they can bring your product from out of reach due to cost to exceeding margin expectations.

The Four Main players

When manufacturing a product, there are four main players that work together to make it successful.

Your company
Your manufacturer
The part’s manufacturer
Everyone in between

This array of ‘middle-men’ is who we are going to look at in this post. Diving into what each does (FAE, Sales Rep, and Distributor) and how they work together to help you bring your product to market.

Field Application Engineers (FAE)

Often the primary contract for engineering teams, the Field Application Engineer (FAE or Apps Engineer) is the technical arm of a company’s sales and marketing team.

While some FAEs may get a commission when your company buys parts, their primary function is to help you get their parts working successfully. This may include supplying you with example code or reference designs to kick start projects or board reviews to get products shipping.

FAEs work for either the part manufacturer themselves or are employees of distributors. Often both are involved at different levels due geographic location. If you happen to be in a tech-focused city, you might be lucky enough to get a few different FAEs to work with, but sometimes it will be a single individual supporting everyone in the state.

The Sales Representatives

This is where it becomes a more complex inter-relationship. The part manufacturer you are buying parts from has a sales team, the distributor (below) has a sales team, and sometimes a third company step in to sell the parts to you. While competition is a normal course of action, it may not be the case where these three groups of sales are competing.

We will go through each of the three below, but at a high level, the sales rep’s job is to have your company buy parts. This is usually good for you, as you are trying to buy parts, but it also comes with unsolicited sales calls, spur of the moment meetings with manufacturers, and other overhead of working with sales.

Part Manufacturer Sales Rep

Every part manufacturer will have a sales team whose primary purpose is to get customers to use their company’s parts rather than another. They represent their employer in all transactions and don’t really care which channel your company buys parts as long as it’s their parts in the end.

A larger part manufacturer may have sales reps for different lines and geography, so you may end up working with multiple sales reps from a manufacturer if you have multiple parts from them.

Distributor Sales Rep

As we will see below, Distributors are the channels through which the parts flow. So the primary goal of distributor’s sales reps is to have you buy parts (really any parts) through them. If you want a certain part, or they suggest a new part, the distributor sales rep will work with the manufacturer sales rep to get the part to you at the best cost everyone can agree on.

The other difference with the distributor sales rep is they manage accounts rather than product lines. So when working with a distributor you will work with a single sales rep that has been assigned to you, even through you buy from multiple manufacturers through them.

Local Sales Rep

The world is a huge place, and engineering companies are growing in nearly every pocket. This makes it hard for part manufacturers to be everywhere needed to sell their product. Distributors are an alternative, but as they often source thousands of lines, including competitors, they may not align with a manufacturer’s best interest.

Instead, they may bring in a small, local company that will represent their business to companies in a smaller geographic area. These companies often represent multiple companies (as shown by their ‘line card’) but they are non-competing companies.

If your company isn’t located near a manufacturer’s sales site, or perhaps just too small of an account for them to spend time on, they will refer you to the local sales rep company, which in turn will bring in a distributor to get the parts to you.

The Distributor

To get parts from the part manufacturer to your product manufacturer, distributor companies come into play. They buy parts directly from thousands of part manufacturers, ship, and store them around the world (concentrated near major manufacturing areas), and then sell the parts to your product manufacturer.

Below are the largest worldwide distributors as of 2013 (via Global Purchasing)

Avnet, Inc. ($25.2 billion)
Arrow Electronics, Inc. ($20.4 billion)
Future Electronics ($7.4 billion)
TTI, Inc. ($1.6 billion)
Allied Electronics, Inc. ($1.5 billion)

When working with Contract Manufacturers or smaller manufacturing companies you will be able to drive which company provides the distribution service based on who is available locally to provide the most FAE and sales support to you.

Other times the manufacturer already has an established relationship with a distributor or even a local parts warehouse in their building managed by a distributor. This limits your options for distributor options.

Because the distributor is where all the money flows, this can become the most competitive group. Sales representatives from other distributors will solicit for your business and when not handled well, it can get ugly.

But it’s also the distributor channel that you can find the most savings for your project. If you carefully designed in alternative parts, the sales group at the distributor can work with the various part manufacturers (remember they distributor for thousands) to get you the best price. They add their service fee on to but you end up with a lower cost product.

Fitting them all together

Bringing it all back together again, some of the mystery and confusion around getting parts to your manufacturer, I hope, is cleared up. Your company wants to buy parts (either specific or whichever is cheapest), get them to your manufacturer, and build a product.

Distributors are the funnel from all the parts manufacturers, sales reps will get you pricing and fight for replacements, and FAEs will show you how to get the parts to work.

While the competition between each of the companies in each group can lead to better pricing and part availability, I have also found that building good relationships with companies that I trust has brought the best results. I have my favorite distributor and FAEs, and I have my favorite silicon vendors – not just because they get the best price, but because we have built a good relationship to help each other’s companies out.

So, while the multiple parties become confusing and inter-jumbled at times, my overall advice is this: be honest and as transparent as possible to each of the companies. If you choose a different part, give them the reasons and see if they can provide a good counter. If your manufacturer uses a different distributor, be honest with the distributor you’ve been working with.

If you want to learn more about bringing a product to market, make sure to sign up for the free Handbook to New Product Introduction below.

NPI in Seven Steps – Engineering Verification

Published by Joe

This is part three of our dive into New Product Introduction, this time with a look at the Engineering Verification stage. If you missed the previous articles, get caught up here.

To this point in the New Product Introduction process, you don’t really have much to show for all the hard work. A couple of reports, pretty renderings of the mechanicals, and lots of hand-waving around the prototype.

Get ready to change all of that. In fact, get ready to become the most popular team in the company, everybody will soon ‘be in the area.’ Why? Because you are about to get the first prototypes from the manufacturer that almost look like a product.

Engineering Verification Run

So now that your manufacturer has reviewed the design, helped you make some changes, and gave their best estimate of the final cost, now it’s time to build something to show for all the hard work.

The very first build at the manufacturer is the Engineering Verification build or EV. Depending on how much money your company wants to spend, this could be as little as 10 units, or as many as a hundred, but it will be a very small number compared to full production – and much more expensive.

Why so little? The manufacturer wants to make sure that after building everything to your specification, that it actually works and is what you wanted before building (and wasting) tons of broken boards.

Even for local manufacturers, where there is no language barrier, interpretations and assumptions between the two teams can cause changes that were unexpected. By actually building a few units, the manufacturer and your team can come to a full agreement by actually being able to touch and hold the tangible product of the assembly.

The First Products

The EV run is probably also the first time the engineering staff have had enough units to test and get user feedback. Marketing and sales inevitably want some to show off, and management will take a few more to present upwards.

This is the first ‘real’ units of your cool new thing, so expect everyone to suddenly get excited about it. For some, seeing renderings of what the product will look like and hold hacked together prototypes really doesn’t sink in for them. But once you place that first, real product in front of them, the light bulb goes off and they get fired up.

While this does contradict the earlier statement that the EV run is fairly small, it does need to be big enough for this sudden demand. So if the engineering team needs 15 boards, make sure you order 30 – and lock up those 15.

What to do with Engineering Units

So beyond verifying with the manufacturer that the build works and providing show-and-tell items, what do you do with the EV units?

Obviously it varies greatly depending on the product and how fast your team is moving, but a handful of items to check off with an Engineering Verification build are:

Verify changes between prototype and EV builds still work. You likely modified a few things from the dev kits so this is to make sure things still work.
Test that manufacturing variations don’t disrupt functionality. A manufacturer’s process has a margin for errors so this is a good first look at how they will affect the performance and look of the product.
Test larger populations communicating to servers. Having one or two devices running is one thing, but now you could have dozens trying to all communicate, adding a bit more stress on the whole system.
Abuse with firmware updates and temperature cycling. Now that there are a couple “extras,” a few might get abused to test to make sure firmware updates will be reliable and temperature doesn’t effect anything.
Initial FCC or other regulatory scans. Because certifications can be a process, the soon the product can be checked, the easier the process will be. You can’t do the full certifications here but you can catch issues early.
Ensure connections and attachments align properly. Comes back to the manufacturing tolerances, but if the product is supposed to connect with other devices you can now ensure everything will connect correctly.

Basically, the Engineering Verification build is to make sure that everything works before taking the next step.

What Happens if it Fails?

Every step of the NPI process is a traffic light. Before moving past the EV build, everyone must sign off. Build or design failures are reasons to cause the project to halt.

If the EV units don’t pass inspection or need design changes everything must be re-analyzed. Not something anyone wants, but it is part of the process and ensures everything moves forward smoothly.

Small changes (switching components) can just be swapped in for the next build and everything can continue. Major changes (like re-spinning the PCB) may cause the project to go back through DFM and EV builds again – delaying the process and causing additional expenses.

But don’t just ignore problems to keep the schedule! It’s much better to find things now than in the next builds. You’ve only built 10 or a hundred boards, it’s nothing compared to the millions everyone wants to build.

So hit the EV units hard and be very confident with the design before moving forward.

Importance of EV for the Manufacturer

Your team isn’t the only group to get value out of the EV build. For the manufacturer, they usually build a couple extra sets for their own uses.

While you are verifying the product is working and showing it off to the CEO, they are preparing their system for the next steps of the NPI process.

Namely, they use these boards for:

Internal documentation and process evaluation
Design test tools and setups (more in the next article)
Debug issues from their end
Have product on hand to help communication

For the manufacturer, having boards in hand is just as critical to their process as it is to yours. So when they request a few more, make sure they get them.

The End Result

At the end of the Engineering Verification stage of the NPI process, your team will have working boards that are functionally close to the final product.

Everyone will be able to hold the actual device and give their opinions, but most of all, get excited that this idea is becoming reality.

In the next article, we will look at how the manufacturer will test and verify the product is built correctly before moving on to the next big build. If you want to get notified of the next articles and get our free Handbook to NPI, add your email below.

Making Mistakes and Learning From Them

Published by Joe

This blog almost didn’t happen. I was afraid I’d make mistakes in what I write or what I say. In fact, many other projects that I have started have completely sputtered out due to this fear of failure.

Have I already made some mistakes? Yup, and I will continue to do so as I go. What I have not done is to allow these mistakes to cripple me. Instead, I allowed myself to use them to improve, and so can you.

Making Mistakes, and Learning From Them

Success is like getting high, but failing is like going to the gym.

When things go right everybody feels good, everybody is happy, and nobody cares why.

When you tied your shoe this morning did you reflect on how it could have been improved? Or on today’s commute, did you think about how to add additional safety processes to the transportation system?

Probably not, because they worked just as expected. Things that work rarely get attention, and therefore rarely get improved.

Now think of something that didn’t go so well this week. The program crashed, late to a meeting, said something embarrassing.

Afterwards did you stop and think about what happened? Did you fix the bug and make the software more efficient in the process? Your boss take you aside to have a chat?

Mistakes get attention, but with that attention, you get to actually improve yourself and your products.

Every mistake is like a push-up, they suck every time, but the more you do the stronger you get.

Show That You Can Learn

When mistakes happen it can sometimes feel like your career is over. But unless your really screwed up (and try to cover it up – aka VW) it’s more likely your team and management are looking at what you’ll do next.

“We learn from failure, not from success!” – Bram Stoker, Dracula

When the company hired you (especially if this was your first career job) they knew they would be investing in you. They want you to produce, yes, but they also want you to learn and grow so that the next project you do is even better.

So when that next mistake comes, remember that your team is looking for how you respond.

Do you blame others, make excuses, and sulk away embarrassed? All bad choices by the way. Instead, take accountability, share your plan to fix the problem, and how you can make changes to prevent it from happening again.

This may garner you more respect and admiration from your teammates than having done the project successfully in the first place.

Your Take Away

Mistakes suck, they make you look bad, and they can cause serious harm or cost. But it’s how you recover that will propel your career.

For your next ‘oops,’ do a postmortem – sit down and analyze what went wrong and how it was allowed to happen. Interview your coworkers to get their take (remember you are trying to learn, not place blame). Play back the events in your head and imagine how it could have gone differently.

Afterwards, present your findings and plan to your manager, admitting your part in the mistake and what you are doing to make up for it.

The fist time it may take a good deal of time to put everything together, and it may feel like a big waste of energy to do for every mistake, but it will become faster as it becomes automatic. Jeff Atwood (Author of Coding Horror) put it: “I don’t think it matters how you conduct the postmortem, as long as you do it.”

Don’t let the fear of making a mistake paralyze you into inaction. Take some risks, learn from each misstep, and improve your methods for the next attempt.