Device onboarding: "certificate already exists"

[teba99]

Hi,

We have onboarded several hundred nrRF9151 devices using the nRF Cloud "bulk onboard" UI (preconnect onboarding using device_credentials_installer.py). Generally, this works fine.

However, for two devices we are getting an error during onboarding: "certificate already exists". But the respective devices (UUID) are not present in nRF Cloud in any of our teams.

We're not sure why this is happening and how to resolve the situation and finish onboarding the two devices.

Any advice is appreciated.

Thanks

[Markku Lehto]

Hi Terrence,

We need to investigate what might be the reason for "certificate already exists" error message. Could you give us device UUIDs for those two devices you had issues with it?

Also, I assume you are using https://github.com/nRFCloud/utils?tab=readme-ov-file#how-to-registering-devices-quickly to register your devices. Also, if you could add those certificates to this ticket as well which caused the error message. 

Thank you.

[teba99]

Hi Markku,

Correct, we are using that method to register our devices. The two devices are 50343956-3037-43c5-8029-001906c2202a und 5034474b-3731-4fb9-8020-0713b6b5484e.

Thanks

[teba99]

Hi,

50343956-3037-43c5-8029-001906c2202a: We verified this is correct and tried onboarding again, but it continues to fail.

5034474b-3731-4fb9-8020-0713b6b5484e: Yes, 25c3f2c9-732a-45c6-a580-c3b0f327af99 is one of our teams, but this device is not listed in the team. So we can't remove it.

Thanks

[Markku Lehto]

Hi,

ok I will removed 50343956-3037-43c5-8029-001906c2202a and 5034474b-3731-4fb9-8020-0713b6b5484e from the our registry again.

If it continues to fail to onboard these devices, could you send those two certificates for these two devices to me? I suspect the issue is related to these certificates.

If you could upload these certificates in PEM format to this ticket, I would be able to check them.

Thank you,

Markku Lehto 

 

[teba99]

Hi,

We retried onboarding, but it still fails. Certificates attached.

Thanks

certs.zip

[Markku Lehto]

Thanks Terrence,

I will troubleshoot this further.

-M

[Markku Lehto]

Hi,

It seems that those two certificates were already registered in our system. For some reason they were orphaned, so no device was attached to them.

I removed both of those certificates which seems to be from some earlier onboarding session. You should be good to go to onboard these two devices now.

In case you still have issues, please reopen this ticket.

Thank you for using nRF Cloud.

Best regards,

Markku Lehto

[teba99]

Markku, 

we now have more devices where this is happening. What we do is:

1. Onboard the devices in nRF Cloud teamA

2. Test and configure the devices in teamA

3. Delete the devices from teamA

4. Onboard the devices in nRF Cloud teamB <-- this is where the problem occurs

It seems, in step 3, for a certain percentage of devices the certificates are not cleanly removed from the system.

Can you please remove the certificates for the following devices. Also, please let us know how we can avoid this issue going forward.

Thanks!

50343956-3037-4123-8037-0f1128bba843

50343956-3037-4693-8090-0c16d6b8bf49

50343956-3037-43c5-8029-001906c2202a

5034474b-3731-4fb9-8020-0713b6b5484e

50343956-3037-411c-8009-091efaf720a7

50343956-3037-4d66-808a-16273b09d886

50343956-3037-48c9-8098-211d15f9aff1

50343956-3037-4563-80df-07227ae0c6d6

50343956-3037-489d-80a0-0a155ddbaf74

[Markku Lehto]

Hi Terrence,

We need to investigate this with the team. What is the time interval between step 3 and step 4? Did you see any errors? Steps 1 and 4 were done using bulk onboarding with csv, did you see any errors? Bulk onboarding is asynchronous call so it might take some time to complete. Could you estimate how many devices were onboarded at the same time?

In the meanwhile, you need to send me all these certificates which were reported as a duplicate.

Thanks,

-M

[teba99]

Time between step 3 and 4: I believe at least several hours, if not days. 

I don't think we saw any errors during step 3. First error was in step 4, for a subset of devices.

I believe we onboarded several dozen devices in step 4. 

Certs attached.

Thankscerts2.zip 

[Markku Lehto]

Hi,

ok, understood. Team will need to investigate this. In the meanwhile I will be removing these certificates for you. I will let you know when they are removed.

Thanks,

-M 

[Markku Lehto]

Hi Terrence,

I checked these devices and deleted orphaned certificates. Some of these devices seem to have onboarded just fine, see the list:

50343956-3037-411c-8009-091efaf720a7 removed orphaned certificate
50343956-3037-4123-8037-0f1128bba843 associated, looks ok
50343956-3037-43c5-8029-001906c2202a associated, looks ok
50343956-3037-4563-80df-07227ae0c6d6 removed orphaned certificate
50343956-3037-4693-8090-0c16d6b8bf49 removed orphaned certificate
50343956-3037-48c9-8098-211d15f9aff1 removed orphaned certificate
50343956-3037-4d66-808a-16273b09d886 removed orphaned certificate
5034474b-3731-4fb9-8020-0713b6b5484e associated, looks ok

I didn't want to do anything for those devices that are with comment "associated, looks ok". If you have issues with them, let me know. For others, I removed orphaned certificate. You can try to onboard them again. I filed a bug to the team and they will start investigating this. Let's keep this ticket open until I have further information to tell about this issue or you bump into this again.

Best regards,

Markku Lehto