AT Command interface hangs

RE: AT Command interface hangs

kt1024 — Thu, 21 Feb 2019 14:46:15 GMT

I'll do that.

RE: AT Command interface hangs

Jan Tore Guggedal — Thu, 21 Feb 2019 08:35:03 GMT

I think it would be good if you created a new, private ticket her on DevZone, for these more complex issues so that the support engineers can provide the best help and you can keep your code confidential.

RE: AT Command interface hangs

kt1024 — Thu, 21 Feb 2019 08:28:46 GMT

I'm working towards getting you access to the repository, I'll have to recreate the EEEEEEEEE code since I scrapped it. I also have another issue now where a socket fails after 15 mins with "Operation not supported on socket" and then hours later the nRF91 resets itself.

The sleep fix has been in my code since you first mentioned it.

Regards,

RE: AT Command interface hangs

Jan Tore Guggedal — Thu, 21 Feb 2019 08:01:15 GMT

And then there's also a possibility that your seeing this error:
https://www.nordicsemi.com/DocLib/Content/Errata/nRF9160_EngA/latest/ERR/nRF9160/EngineeringA/latest/anomaly_160_17

You can apply the workaround described in the document and see if that helps with your issues.

RE: AT Command interface hangs

Jan Tore Guggedal — Thu, 21 Feb 2019 07:36:23 GMT

Hi,

I've never seen that. It's really curious.

It would be very useful for both others and me to see a repository or something with your codebase so that we can reproduce locally and get to the bottom of this.

One thing you could try just to eliminate one thing, is to include a k_sleep(100) or k_sleep(10) in the bsd_os_timedwait function i mentioned earlier. That might help narrow things down a bit.

BR,
Jan Tore

RE: AT Command interface hangs

kt1024 — Wed, 20 Feb 2019 09:58:16 GMT

I found some time. I tried to setup a thread that creates the socket to the AT command interface, reads and send bytes to the AT socket and fills and empties a queue for rx and tx bytes that connects with the main task .While I can read from the socket it only ever returns 'E' characters

Line1: EEEEEEEEE

This result came about when I sent "AT+CGSN=1\r\n". This happens only once. I didn't use any mutexes since I expected the queues to be sufficiently protected and the socket was only used in the read/write task.

If you have seen this before let me know, otherwise I will get rid of the threadding and just don't close the socket in lte_lc.c

Regards,

RE: AT Command interface hangs

kt1024 — Fri, 15 Feb 2019 12:48:37 GMT

Unfortunately I don't have time next week. So I'll have to let the case linger for a bit. Also since at_host behaves much better and I strongly believe that since recv() is meant to be called with way more error handling and retries anyway, I would rather change my code to something like at_host has to offer.

On the other hand if you think that there might be some issue with the modem or the binary code in nrfxlib I would be happy if it can become more stable.

Regards,

RE: AT Command interface hangs

Jan Tore Guggedal — Fri, 15 Feb 2019 12:20:01 GMT

Bummer, then I'm starting to run out of ideas, but will come back to you if I can find something. If your code is on a public git branch somewhere, that would be great, to reproduce it exactly as you have it locally. If you don't want to have it public, you can open a private ticket and share your code only with tech support.

RE: AT Command interface hangs

kt1024 — Fri, 15 Feb 2019 12:00:32 GMT

I tried this, it doesn't prevent recv() from hanging.

Regards,

RE: AT Command interface hangs

Jan Tore Guggedal — Fri, 15 Feb 2019 11:41:37 GMT

Thanks for the log output! I was able to reproduce locally, and it seems you've hit a bump with regards to missing implementation of the function bsd_os_timedwait(). Could you try this hack in nrf/lib/bsdlib/bsd_os.c :

int32_t bsd_os_timedwait(uint32_t context, uint32_t timeout)
{
	    /* TODO: to be implemented */
    	k_sleep(timeout);
	    return 0;
}

This is not a long term solution, but it can help you overcome this weird issue for now and isolate the issue.

RE: AT Command interface hangs

kt1024 — Fri, 15 Feb 2019 10:37:03 GMT

Hi,

I tried instrumenting the code around socket() and found that the socket descriptor never changes its value and stays at 2, also it never reaches return -EFAULT.

int exl_lte_lc_get_cereg(exl_lte_lc_cereg_t * const pCereg)
{
    int errVal;
    int at_socket_fd;
    u8_t buffer[LC_MAX_READ_LENGTH];

    at_socket_fd = socket(AF_LTE, 0, NPROTO_AT);
    if (at_socket_fd == -1)
    {
        printk("%s:%u: Error EFAULT\n",__FUNCTION__,__LINE__);
        return -EFAULT;
    }

    printk("%s:%u: Socket: %d\n",__FUNCTION__,__LINE__,at_socket_fd);

    errVal = at_cmd_wret(at_socket_fd, cereg, AT_CMD_SIZE(cereg), buffer);

    if (errVal == 0L)
    {
        // "+CEREG: 2,5,\"C76A\",\"01ACB906\",9"

        sscanf(&buffer[8], "%u,%u,\"%[^\"]\",\"%[^\"]\",%u", (unsigned int *)&pCereg->n, (unsigned int *)&pCereg->stat, pCereg->tac,
                pCereg->ci, (unsigned int *)&pCereg->act);

    }
    else
    {
        return errVal;
    }

    close(at_socket_fd);
    return 0;
}

There is a high consistency in reaching the 140 seconds mark and not getting further. Also it hangs in at_cmd_wret():recv() and not elsewhere. All this lets me think that awneil is on the right track, despite my sloppy coding involving not checkng return values and returning from a function in more than one place.

Let me add a bit of output from UART_0:

s
exl_lte_lc_get_cesq:312: Socket: 2
Before recv
After recv
rx1: +CESQ: 99,99,255,255,255,61
rx2: OK

rx3: +CESQ: 99,99,255,255,255,61
352656100000036?b=[s="Test",c=01ACB906,tc=C76A,r=-79.0,ti="127",a=0]Sent 68 Bytes
352656100000036?b=[s="Test",c=01ACB906,tc=C76A,r=-79.0,ti="137",a=0]Sent 68 Bytes
exl_lte_lc_get_cereg:247: Socket: 2
Before recv
After recv
rx1: +CEREG: 0,5,"C76A","01ACB906",9
rx2: OK

rx3: +CEREG: 0,5,"C76A","01ACB906",9
352656100000036?b=[s="Test",c=01ACB906,tc=C76A,r=-79.0,ti="147",a=0]Sent 68 Bytes
exl_lte_lc_get_cesq:312: Socket: 2
Before recv

Thanks for the update about at_host.c I'm using this in its older form already. Using a separate thread for reading the interface is something I may also try in the future.

Regards,

RE: AT Command interface hangs

Jan Tore Guggedal — Thu, 14 Feb 2019 16:21:24 GMT

Hi,

I think the problem is on line 25 in the middle code block you posted. In case of return, the socket remains open, but a new one is opened in the next run. In the end you run out of available sockets, and I guess you would see -EFAULT returned from exl_lte_lc_get_cereg(). You can either just keep the socket open all the time, or ensure that you close it also when something fails.

Update: You can see the new implementation of the AT host for tips on how you can send and receive AT commands in a dedicated thread, and receive callbacks whenever there's data on the socket: https://github.com/NordicPlayground/fw-nrfconnect-nrf/blob/master/lib/at_host/at_host.c
This allows you to receive notifications for example when the signal strength changes, without having to poll the modem, using the Nordic proprietary AT command AT%CESQ: www.nordicsemi.com/.../proc_cesq

RE: AT Command interface hangs

kt1024 — Thu, 14 Feb 2019 06:43:00 GMT

Well the tags are:

\zephyr>git log
commit 82fe1f9dfcf0e100d6fc952c07cf63d63aa08a0e (HEAD, tag: v1.13.99-ncs2-rc2, tag: v1.13.99-ncs2)

\nrf>git log
commit 5ad94691363cb0832943d8cb759df7db982a2e63 (HEAD, tag: v0.3.0)

\nrfxlib>git log
commit 891c089577857d8a197b7360e007a0d84e6dbe54 (HEAD, tag: v0.3.0)

any newer versions would be great to have, I just went through the Getting Started App for this.

The modem firmware is:

mfw_nb1_nrf9160_0.3.0-73.prealpha.zip

The main change to lte_lc.c is:

static int at_cmd_wret(int fd, const char *cmd, size_t size, char * const ret)
{
    int len;
    uint32_t retCount = 0;
    int fnc_return;
    u8_t buffer[LC_MAX_READ_LENGTH];

    char *pLine[2] =
    { NULL, NULL };

    LOG_DBG("send: %s", cmd);
    len = send(fd, cmd, size, 0);
    if (len != size)
    {
        LOG_ERR("send: failed");
        return -EIO;
    }

    len = recv(fd, buffer, LC_MAX_READ_LENGTH, 0);

    memset(ret,0,LC_MAX_READ_LENGTH);
    do
    {
        if (len < AT_CMD_SIZE(success))
        {
            LOG_ERR("recv: %s", buffer);
            fnc_return = -EIO;
        }
        else
        {
            pLine[0] = buffer;
            pLine[1] = memchr(buffer, '\n', LC_MAX_READ_LENGTH);
            pLine[1][0]='\0';
            pLine[1]++;
            if (memcmp(success, &pLine[1][0], AT_CMD_SIZE(success)) == 0)
            {
                memmove(ret, pLine[0], strlen(pLine[0]));
                fnc_return = 0L;
            }
            else
            {
                LOG_ERR("recv: %s", buffer);
                fnc_return = -EIO;
            }

        }

        printk("rx1: %s\n", pLine[0]);
        printk("rx2: %s\n", pLine[1]);
        printk("rx3: %s\n", ret);
        retCount++;
    }
    while (retCount < 1U);

    return fnc_return;
}

and to give an example of using this function:

int exl_lte_lc_get_cereg(exl_lte_lc_cereg_t * const pCereg)
{
    int errVal;
    int at_socket_fd;
    u8_t buffer[LC_MAX_READ_LENGTH];

    at_socket_fd = socket(AF_LTE, 0, NPROTO_AT);
    if (at_socket_fd == -1)
    {
        return -EFAULT;
    }

    errVal = at_cmd_wret(at_socket_fd, cereg, AT_CMD_SIZE(cereg), buffer);

    if (errVal == 0L)
    {
        // "+CEREG: 2,5,\"C76A\",\"01ACB906\",9"

        sscanf(&buffer[8], "%u,%u,\"%[^\"]\",\"%[^\"]\",%u", (unsigned int *)&pCereg->n, (unsigned int *)&pCereg->stat, pCereg->tac,
                pCereg->ci, (unsigned int *)&pCereg->act);

    }
    else
    {
        return errVal;
    }

    close(at_socket_fd);
    return 0;
}

The main loop looks like the following:

int main(void)
{
    int32_t s;
    char imei[16];
    uint8_t stage = 0U;
    char dgram[256];
    int32_t Rsrp;
    uint32_t uptime;
    uint32_t vbat;
    exl_lte_lc_cereg_t Cereg;


    printk("Application started\n");

    memset(dgram,0,sizeof(dgram));
    memset(&Rsrp,0,sizeof(Rsrp));
    memset(&uptime,0,sizeof(uptime));
    memset(&vbat,0,sizeof(vbat));
    memset(&Cereg,0,sizeof(Cereg));
    memset(imei,0,sizeof(imei));

    buttons_leds_init();
    work_init();
    modem_configure();

    s = udp_connect();

    if (IS_ENABLED(CONFIG_CLOUD_UA_CONSOLE))
    {
        console_init();
    }

    // get imei
    exl_lte_lc_get_imei(imei);

    while (true)
    {
        if (s >= 0L)
        {
            uint32_t slen;


            uptime = (uint32_t)(k_uptime_get() / 1000L);


            switch (stage)
            {

            case 0:
                stage=0U;
                break;
            case 1:
                exl_lte_lc_get_cereg(&Cereg);
                stage=2U;
                break;
            case 2:
                exl_lte_lc_get_cesq(&Rsrp);
                stage=0U;
                break;
            case 3:
                exl_lte_lc_get_vbat(&vbat);
                stage=0U;
                break;

            default:
                stage=0U;
            }

            sprintf(dgram, "%s?b=[s=\"Test\",c=%s,tc=%s,r=%.1f,ti=\"%u\",a=%u]", imei, Cereg.ci, Cereg.tac,
                    (float)1.0f * Rsrp, (unsigned int)uptime, (unsigned int)((100 * vbat) / 5000U));

            printk("%s", dgram);

            slen = nrf_strnlen(dgram, sizeof(dgram));
            if (send(s, dgram, slen, 0L) != slen)
            {
                printk("partial/failed write\n");
            }
            else
            {
                printk("Sent %u Bytes\n", (unsigned int)slen);
            }


        }
        /* if logging is enabled, sleep */
        k_sleep(K_MSEC(10000));

    }

    return 0L;
}

Should I change the approach in exl_lte_lc_* I'll go with the one used in at_client. For now I don't need to use AT_commands in the main loop.

Regards,

RE: AT Command interface hangs

Jan Tore Guggedal — Wed, 13 Feb 2019 18:47:43 GMT

Hi kt1024,

Could you share the changes that you've made in lte_lc.c/h and how you call the functions?
It would also be good to know which versions of the SDK your working on, for example the latest commits of the -zephyr and -nrf repositories in particular, along with modem version. That way others can replicate what your setup and possibly provide better help.

Best regards,

Jan Tore

RE: AT Command interface hangs

kt1024 — Wed, 13 Feb 2019 14:51:36 GMT

Appearently I changed from proactive to grumpy. In conclusion I would say that I have no problem switching to better socket handling than is done now. Since this is only an evaluation project I could also do without polling, but there are enough good reasons to use it in the end product to not yield on this.

Thanks for your help,

RE: AT Command interface hangs

awneil — Wed, 13 Feb 2019 14:16:31 GMT

[quote userid="76865" url="~/f/nordic-q-a/43620/at-command-interface-hangs/170917"]I expressly do not think about the impact on the modem[/quote]

I was referring to your opening question:

[quote userid="76865" url="~/f/nordic-q-a/43620/at-command-interface-hangs"]how I can prevent the AT command interface hangup[/quote]

which seemed to be a "modem impact" thing.

I think I've lost track of where you're at/going - if I ever understood it in the first place.

RE: AT Command interface hangs

kt1024 — Wed, 13 Feb 2019 14:09:12 GMT

I expressly do not think about the impact on the modem. How would you treat an output buffer overrun ? Is there any reason not to just empty it and dump more stuff in. There are perfectly good ways of being responsive about it. There is no reason to go into a corner and sulk forever.

Regards,

RE: AT Command interface hangs

awneil — Wed, 13 Feb 2019 11:37:49 GMT

[quote userid="76865" url="~/f/nordic-q-a/43620/at-command-interface-hangs/170837"]polling is ok for me[/quote]

I was thinking of the impact on the modem.

[quote userid="76865" url="~/f/nordic-q-a/43620/at-command-interface-hangs/170837"]if URCs are your only means of getting information from the modem, then you can't decide whether a process on the modem isn't working or whether it just has no reason to generate a URC.[/quote]

Fair point.

Maybe a combination of URCs and the occasional poll as a "sanity check" ... ?

RE: AT Command interface hangs

kt1024 — Wed, 13 Feb 2019 11:32:07 GMT

If I run my own lte_lc.? this could work, but I've had bad experiences with URCs elsewhere so polling is ok for me. I could imagine that powerconsumption and used interface bandwidth could be reduced if polling is avoided but sometimes I just have to get a piece of information at a certain time.

Another issue arises if URCs are your only means of getting information from the modem, then you can't decide whether a process on the modem isn't working or whether it just has no reason to generate a URC.

So while I'm at it, if you have a flakey modem only providing URCs and no means to poll for that information is a bad idea.

Regards,

RE: AT Command interface hangs

awneil — Wed, 13 Feb 2019 10:45:59 GMT

Rather than keep polling the interface, can you just use unsolicited responses ... ?

RE: AT Command interface hangs

kt1024 — Wed, 13 Feb 2019 10:17:02 GMT

I tried the at_client example with TeraTerm and AT%XVBAT for a while and I couldn't hang up the modem yet so it is capable of answering AT commands for longer than 2 minutes. So there is a way to make it work properly. Ok I guess I know enough for now lets close this issue, the modem reset is worth its own question.

RE: AT Command interface hangs

kt1024 — Wed, 13 Feb 2019 07:02:06 GMT

Well that makes sense, I could just port one of the multiple thousand lines of state machine code we have for our other projects to really pay attention to the modem. That state machine would probably run but it also has the ability to reset the modem - just in case.

What I should do, is scrap the lte_lc.? code that just writes something, reads without paying much attention and closes the interface socket again, and replace it with a reader writer task based on nonblocking sockets and fill the results into a queue. That should present a nice facade of attention and caring to the modem.

I don't feel like I'm being paid to do this right now so it will have to wait until later.

Regards,

RE: AT Command interface hangs

awneil — Tue, 12 Feb 2019 15:00:38 GMT

The commonest mistake people make with AT commands is failing to wait for and pay attention to the responses from the modem - instead just relying upon blind delays:

https://www.avrfreaks.net/comment/2209136#comment-2209136

https://www.avrfreaks.net/comment/2139451#comment-2139451