Time out error while production test using DTM

RE: Time out error while production test using DTM

Håkon Alseth — Wed, 18 Sep 2019 07:52:44 GMT

Sounds like there's something hanging on either the DUT or the tester-side, that requires a delay between tests. I am sorry, but I cannot say anything more in detail here, as there are components here that are out of scope to the products that we deliver. The python scripts delivered with nAN-034 are developed by us, but once you build a GUI on top, it becomes a lot more complex in terms of whats happening in the chain of communication. I'm not stating that the python scripts are waterproof in all scenarios, but I'm just pointing out the complexity compared to running a 10 line python script for one test.

Has the setup behaved like this before with other devices? If your setup has not been behaving this way with other production runs prior to this, it may point to the firmware running on the DUT misbehaving.

Kind regards,

Håkon

RE: Time out error while production test using DTM

Cimteker — Tue, 17 Sep 2019 13:03:33 GMT

Thanks Håkon for your response.

All product samples exhibit this behavior, I have 10 samples.

Could please make some comment about timing for this test? Do I need to wait between switching channels? I found that having 300-500 ms there helps. There was also another delay absolutely needed to make it work. Here's where I inserted it:

- Start transmitting on the requested channel (i.e dev kit)

- Receive packets for 100 ms (DUT)

- Wait 0.5 sec

- Stop Transmitting on the requested channel (dev kit)

Without this 0.5 second delay, it would not work. We'd be getting exceptions, meaning no response to LE_TEST_END command.

If I increase the receiving window to 1 s (from 100 ms), this 0.5 delay is not necessary.

Is there an explanation for it?

RE: Time out error while production test using DTM

Håkon Alseth — Tue, 17 Sep 2019 07:58:17 GMT

Looking through the code, there is a theoretical chance that this is related to receiving a framing error on the nRF-side, and if you are running based on the python scripts in nAN-034, these will not retry commands by default. However, if this was the case, the result should be a timeout on the command itself, not a high packet error.

Are these issues only observed when testing against one specific DUT, or is this something that you have seen on other products as well?

Kind regards,

Håkon

RE: Time out error while production test using DTM

Cimteker — Mon, 16 Sep 2019 15:10:48 GMT

Hi Håkon, thanks for your reply.

We do not have any other hardware. We're a system integration house, we only buy what we need to complete the project, no spares.

I think my communication is ok. I do get occasional timeouts on the DUT side, but I do not know what firmware they are running.

Here's a rough sketch of what our system looks like, hopefully, this will give you a better idea.

RE: Time out error while production test using DTM

Håkon Alseth — Mon, 16 Sep 2019 14:39:34 GMT

Do you have a nRF52 kit for instance? It would be interesting to see if this issue still occurs on other devices as well.

Could be a problem with the UART handling, where the firmware gets a framing error or similar. The default behavior for the nRF dtm firmware is to assert (softreset) if this happens. Have you tried with a different USB-UART bridge?

Kind regards,

Håkon

RE: Time out error while production test using DTM

Cimteker — Fri, 13 Sep 2019 13:36:58 GMT

Hi Håkon, thanks for your reply.

I use LE_TEST_END for each channel, the power for the dev kit is not cycled during testing.

I do not have another dev kit to try. It ran overnight with 1400 cycles and 4 failures so that's not bad, but still.

RE: Time out error while production test using DTM

Håkon Alseth — Fri, 13 Sep 2019 07:16:04 GMT

That is strange. How does your test setup handle the nRF between tests? Are you powering the device down and up again, or are you sending LE_TEST_END then starting a new one?

Have you tried replacing the kit in the tester to see if this has any impact on the problem?

Kind regards,

Håkon

RE: Time out error while production test using DTM

Cimteker — Thu, 12 Sep 2019 15:04:06 GMT

Thank you Håkon,

I have the latest firmware running now. There's still this problem with occasional failure. 1-2% of the time it will read one of the channels very high ( >40%). I am struggling to explain this. Any ideas?

RE: Time out error while production test using DTM

Håkon Alseth — Thu, 12 Sep 2019 13:51:32 GMT

the one's marked _s130.hex expect that the softdevice is loaded into the device, meaning; the application is offset in flash and RAM.

RE: Time out error while production test using DTM

Cimteker — Thu, 12 Sep 2019 13:17:06 GMT

Thank you Håkon,

What is the difference between pca10028.hex and pca10028_s130.hex files?

RE: Time out error while production test using DTM

Håkon Alseth — Thu, 12 Sep 2019 06:58:59 GMT

It does indeed look like you're using the default pins.

You drag the file "\nRF5_SDK_12.3.0_d7731ad\examples\dtm\direct_test_mode\hex\direct_test_mode_pca10028.hex" to the mass storage device labeled "J-LINK" and that's it. It should boot up within seconds. You can power cycle if you'd like, but wait atleast 4-5 seconds after drag-and-dropping the .hex file.

Kind regards,

Håkon

RE: Time out error while production test using DTM

Cimteker — Wed, 11 Sep 2019 15:25:29 GMT

Hi Håkon, how do you load the precompiled hex file onto the board? I see a section that mentions loading it to the board USB drive. What happens next? Does it load after power toggle?

RE: Time out error while production test using DTM

Cimteker — Wed, 11 Sep 2019 14:54:36 GMT

Hi Håkon, thanks for your response. I think I am using the default RXD/TXD pins.

Could you confirm from the picture?

RE: Time out error while production test using DTM

Håkon Alseth — Wed, 11 Sep 2019 13:53:42 GMT

The documentation on how to get started is located here:

https://infocenter.nordicsemi.com/topic/com.nordic.infocenter.sdk5.v12.3.0/nrf51_getting_started.html?cp=5_5_7_1

It does not look like you're using the default RXD/TXD pins (P0.09 and P0.11; in which you could have used the precompiled .hex file), so it looks like you have to change the pins in main.c::uart_init().

Here's how the function looks now:

static void uart_init(void)
{
    // Configure UART0 pins.
    nrf_gpio_cfg_output(TX_PIN_NUMBER);
    nrf_gpio_cfg_input(RX_PIN_NUMBER, NRF_GPIO_PIN_NOPULL);

    NRF_UART0->PSELTXD       = TX_PIN_NUMBER;
    NRF_UART0->PSELRXD       = RX_PIN_NUMBER;
    NRF_UART0->BAUDRATE      = DTM_BITRATE;

    // Clean out possible events from earlier operations
    NRF_UART0->EVENTS_RXDRDY = 0;
    NRF_UART0->EVENTS_TXDRDY = 0;
    NRF_UART0->EVENTS_ERROR  = 0;

    // Activate UART.
    NRF_UART0->ENABLE        = UART_ENABLE_ENABLE_Enabled;
    NRF_UART0->INTENSET      = 0;
    NRF_UART0->TASKS_STARTTX = 1;
    NRF_UART0->TASKS_STARTRX = 1;
}

These defines comes from the board definitions, so its easier just to hard-code it in main, like this:

#define MY_TXD_PIN XX
#define MY_RXD_PIN YY

static void uart_init(void)
{
    // Configure UART0 pins.
    nrf_gpio_cfg_output(MY_TXD_PIN);
    nrf_gpio_cfg_input(MY_RXD_PIN, NRF_GPIO_PIN_NOPULL);

    NRF_UART0->PSELTXD       = MY_TXD_PIN;
    NRF_UART0->PSELRXD       = MY_RXD_PIN;
    NRF_UART0->BAUDRATE      = DTM_BITRATE;

    // Clean out possible events from earlier operations
    NRF_UART0->EVENTS_RXDRDY = 0;
    NRF_UART0->EVENTS_TXDRDY = 0;
    NRF_UART0->EVENTS_ERROR  = 0;

    // Activate UART.
    NRF_UART0->ENABLE        = UART_ENABLE_ENABLE_Enabled;
    NRF_UART0->INTENSET      = 0;
    NRF_UART0->TASKS_STARTTX = 1;
    NRF_UART0->TASKS_STARTRX = 1;
}

If you run into issues, just let me know what TXD/RXD pins you're using, and I can compile one for you.

Kind regards,

Håkon

RE: Time out error while production test using DTM

Cimteker — Wed, 11 Sep 2019 13:34:15 GMT

Thank you Håkon for your suggestion.

Can you point me to documentation describing the process of loading firmware? I joined the project late and the original engineer has left our company.

RE: Time out error while production test using DTM

Håkon Alseth — Wed, 11 Sep 2019 11:17:24 GMT

Hi,

This is strange test results.What you could check is that you are using the latest direct_test_mode firmware for the nRF51x22 device, and see if this issue behaves differently if you use the example from SDK v12.3 (path is \nRF5_SDK_12.3.0_d7731ad\examples\dtm\direct_test_mode):

http://developer.nordicsemi.com/nRF5_SDK/nRF5_SDK_v12.x.x/nRF5_SDK_12.3.0_d7731ad.zip

This is the latest SDK that has support for nRF51-series devices.

Kind regards,

Håkon

RE: Time out error while production test using DTM

Cimteker — Tue, 10 Sep 2019 13:59:42 GMT

I kept the original limits from the example. 30 % or less is a failure. My script calculates error rate by dividing the number of received packets by the maximum theoretical number (1600 in my case).

You can see from the screenshot provided that channel 37 fails 3 times with 46,47 and 45% error rate.

You can also see in the window below the actual number fo packets received 869,846,880.

I am testing 9 channels: 1,2,3,21,22,23,37,38,39.

So, in summary, I am getting occasional failures when it fails 3 times in a row. Sometimes the part will fail 2 times in a row and then pass another 100+ times.

The numbers you see in the example provided are with higher attenuation levels so I can show the failures. Normally, when I lower the attenuation, most of the results (error rates) are 0 and then the occasional failure.

RE: Time out error while production test using DTM

Cimteker — Tue, 10 Sep 2019 13:50:55 GMT

RE: Time out error while production test using DTM

Edvin — Tue, 10 Sep 2019 06:49:42 GMT

[quote user="Cimteker"]when it fails, it fails 3 times in a row[/quote]

So what exactly is it that fails? Do you loose 3 packets in a row?

Do you get some sort of log file that is generated when you run the test? If so, can you upload it here? I am struggling a bit to understand what exactly is failing.

RE: Time out error while production test using DTM

Cimteker — Mon, 09 Sep 2019 13:33:44 GMT

About 1600. My runtimeinms is 1000 and the length is 10

RE: Time out error while production test using DTM

Edvin — Mon, 09 Sep 2019 12:00:56 GMT

How many packets are you sending on each channel?

RE: Time out error while production test using DTM

Edvin — Mon, 09 Sep 2019 12:00:30 GMT

How many packets are you sending on each channel?

RE: Time out error while production test using DTM

Cimteker — Fri, 06 Sep 2019 15:01:55 GMT

Thanks Edvin. The timeout issue is gone once I changed the test time to 1000 ms.

I am still dealing with infrequent failures. I even implemented retries but when it fails, it fails 3 times in a row. I am not sure what could be causing this, my DUT is in a shielded RF box so it cannot be an external interference.

RE: Time out error while production test using DTM

Edvin — Fri, 16 Aug 2019 11:48:41 GMT

[quote user="Cimteker"]Right, but I do not have another DK.[/quote]

How many packets are you sending per channel? Can you try to increase it to something like a 1000 packets per channel? And you are probably also generating a report that you are looking at. Can you send the report from a test after you increased the number of packets?

Best regards,

Edvin

RE: Time out error while production test using DTM

Cimteker — Wed, 14 Aug 2019 20:09:01 GMT

Right, but I do not have another DK.