My version of this program is now in my github directory.
I’ve ordered some LDOs from element14 to try out.
Also ordered some small (and cheap) MSP430’s to look at an external low power RTC/Watchdog. Even if we resolve the non-booting issue this may still be useful.
Another question I have. I ran a test with the console set to 74k so that I can see the boot messages. It seems to have dies after switching to the other speed, However, the only difference between att the reboots is this line:
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,1)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,0)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,7)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,7)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,0)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
ets Jan 8 2013,rst cause:2, boot mode:(3,7)
ets Jan 8 2013,rst cause:2, boot mode:(3,0)
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
What is this boot mode and why does it differ? According to this
we have
In the boot-up message ‘boot mode:(x,y)’ three low bits of x are {MTDO, GPIO0, GPIO2}. I think that MTD0 is GPIO15? I have 15=LOW, 0&2=floating.
But what is the ‘y’?
Interesting references;
http://www.esp8266.com/viewtopic.php?f=18&t=1418
http://www.eevblog.com/forum/projects/esp8266-'zombie'-very-occasional-boot-failures/
I built a new fw, and a tiny program (uploaded) that just reboots to test this fix.
[update] still hangs after 51 reboots…
Then again after 66…
So we know what is showing at 9600 baud: my own print log, like
> node.restart()
Š0ŠŠŠŠ,ILŠŠDŠ=ŠdŠ$ŠŠCŠ
NodeMCU 0.9.5 build 20150210-1846-eyal powered by Lua 5.1.4
run 1
> dŠ$ŠŠŠ
BŠŠ9$Š
NodeMCU 0.9.5 build 20150210-1846-eyal powered by Lua 5.1.4
run 2
> ŠŠBŠŠŠŠ"ŠŠ)ŠŠ
[trimmed]
NodeMCU 0.9.5 build 20150210-1846-eyal powered by Lua 5.1.4
run 75
ŠŠŠ@ŠŠ1ŠŠ-Š
[hang here]
And at 74880: boot messages like
ets Jan 8 2013,rst cause:2, boot mode:(3,7)
load 0x40100000, len 29548, room 16
tail 12
chksum 0x8b
ho 0 tail 12 room 4
load 0x3ffe8000, len 2872, room 12
tail 12
chksum 0x0f
ho 0 tail 12 room 4
load 0x3ffe8b40, len 13468, room 12
tail 0
chksum 0x05
csum 0x05
ŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠ
[hang here]
However when set to 115200, at hang time one sees the signal in the noise:
Fatal exception (29):
epc1=0x40222788, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000042, depc=0x00000000
[hang here]
Now we know. But we do not understand :-(
My fw build from this morning (now uploaded as 20150213-1100-eyal) seems to be more stable.
[later] After more that an hour of stable reboots I had a look at the bootup messages (@74880 baud) and I now see that it is always showing a stable boot mode (and other items also changed):
ets Jan 8 2013,rst cause:2, boot mode:(3,7)
load 0x40100000, len 29708, room 16
tail 12
chksum 0x30
ho 0 tail 12 room 4
load 0x3ffe8000, len 2852, room 12
tail 8
chksum 0xe0
load 0x3ffe8b30, len 13280, room 0
tail 0
chksum 0x7c
csum 0x7c
It used to show
ets Jan 8 2013,rst cause:2, boot mode:(3,0) <<< last number was any of 0/1/6/7
load 0x40100000, len 29548, room 16
tail 12
chksum 0xb3
ho 0 tail 12 room 4
load 0x3ffe8000, len 2872, room 12
tail 12
chksum 0xc6
ho 0 tail 12 room 4
load 0x3ffe8b40, len 13464, room 12
tail 12
chksum 0x99
csum 0x99
But more importantly, @115200 baud I now see (constantly)
QMEM CHECK FAIL!!!
Whatever it means…
Getting back on-subject. I, for the last few days, am looking at the relationship between power and performance.
I had a very unstable system when using the usual breadboard attached 5v/3.3v device. The setup would work OK for a while, but then fail. The failure is the usual one where a re-flash is required. I think that this is due to the chip getting into some programming mode and overwrite some of the firmware.
I am now using an adjustable DC-DC and will let it run for a long time (a whole day) to check the stability. Power is from a string wall adapter (says 2100mA). The current run is set for 3.3v and shows this:
Flashing 3.292v
Idle 3.300v
Sleeping 3.360v
Running 3.29-3.30v
Reported 3.31v
Only Rx, Tx and GND are attached from the dongle, so no power is supplied.
The “Reported” voltage is the readvdd33() taken at publish time, and is included in the published data. The other voltages are taken with a DMM.
Interestingly, at the first run after a reset the reported voltage was 2.17v, after which is went up to 3.23v-3.25v.
Removing the dongle makes the chip hang. I wonder if the UART buffers fill up? across restarts? I now run with all printings turned off.
I get these readings:
Sleeping 3.361v
Running 3.24-3.28v
Reported 3.17-3.30v
Do we have an instrument that can show the voltage/current across a run (say 10s with ms resolution)?
Do we have an instrument that can show the voltage/current across a run (say 10s with ms resolution)?
Yes, I’ve done this. I’ve got a scope that can do it without problem using a low resistance shunt. Tx pulses were 400mA peak.
Very good. I want to see the full graph during a run. Can you bring in the scope? This should be both educational and fun!
I’ll try. I’m not going to be able to hang around for long tonight.
I am now doing some long tests where I isolate the steps.
All do a dsleep(3*1000000) between runs.
-
An empty loop (just the dsleep). Stopped after about 11,000 runs (10 hours overnight). So I call this stable.
-
Only read the ds18b20 (no WiFi at all). Stopped after 3,800 runs (3h38m). Looks stable too.
-
A full run (read & publish) but without WiFi setup, doing only wifi.sta.connect(), which also saves 1.5s. I suspect that the setup writes to flash, hence this test. Running… Stopped after 3,000 runs. Looks stable again.
This version now uploaded to ESP8266Lib as examples/eyal/ds18b20, but I continue the tests.
-
Following on the suspicion in [3], now do a simple run that only makes a WiFi connection but no reading and no communication. First did a full blank+flash+upload+compile. Running… died in 3 minutes. Case closed?
-
A full run as [3] again, to confirm that it can run for a full day. Running since 18:24… died after 9,421 runs (12h). Good but not perfect.
- This failure is different, the fw and fs were intact, but the WiFi setup was lost (so IP was never acquired). Restarting with
use_old_WiFi_setup = false; dofile ("init.lua")continued the run. Running… failed after another 7,659 runs (9h30). This time it needed a fw flash.
-
Time to run
reBoot.lua[as (1) above] for a longer time. Running… over 25,000 (more than one day). Stopped -
Move to running the exact same test but with WiFi set up (so it will be automatically established). Running… over 25,000 runs (more than one day), stopped.
-
Now running
reWiFi.lua(withuse_old_WiFi_setup = true) which just waits for an IP. Running… 26,000 runs (more than one day). Stopped. -
Now running
reWiFi.lua(withuse_old_WiFi_setup = false). Expecting an early failure. Running… died in minutes.
Tried again… failed soon.
Following up on my experiments so far.
After adding a large cap (470uf) across the power lines, adding pull up/downs wherever it is recommended and using a capable DC-DC off a USB wall charger, I now have it running in the bedroom since 2015/02/25-09:30:02 (yes, I have the log). I did get a few issues along the way but since 2015/02/27-21:14:25 (so about for days now) it is going uninterrupted.
Interestingly, a few times the tmr.now() function returned Infinity, even for the call that is the first line in init.lua.
The following log columns are: time , run count, times, VDD, temp.
The times are : init start, time to read the ds18b20, time as dsleep(60) is issued (so the full run time).
20150303092547 8171 times=0.287835,0.035485,1.2434 vdd=3.38 21.1875 20150303092647 8172 times=Infinity,0.035489,1.24346 vdd=3.38 21.25 20150303092747 8173 times=0.292882,0.03549,1.26849 vdd=3.38 21.25
One small change in the program [in response to test (5) above which I encountered again] is that if the WiFi connect is not completed in 15 seconds (it usually takes a fraction of a second) the program issues the full WiFi setmode/config/connect (but only once). Normally it skips this and just waits for an IP.
An example of a case where (I assume) this was triggered:
20150303081153 8097 times=0.291779,0.035497,1.24335 vdd=3.38 20.375 20150303081256 8098 times=0.285323,0.03549,4.268 vdd=3.38 20.375 20150303081413 8099 times=0.290115,0.03549,19.4173 vdd=3.38 20.375 20150303081511 8100 times=0.286637,0.03549,1.26844 vdd=3.38 20.375
The 8098 line shows a slow cycle (happened a few times) and the following one is probably a failure followed by a reset.
[much later] The setup has now run for over two weeks, more that 25,000 cycles (60s each). It feels so wrong but I plan to experiment further by reducing the supply voltage (it is using an adjustable DC-DC from a wall USB charger) to see how it affects the operation. Now it reports about 3.39v when active, and I expect that it goes up to at least 3.5v when sleeping.
[way later] After running for over a month I did turn it off and tried a few things. It never worked for more that a few hours at a time since :-(
Thanks very much for the experimental testing of this issue. Time for 1S battery + USB charged 150g robots. If I can solve the delay issues with the control interface I have setup.
Steve
So Eyal,
I take it we still don’t know for sure what’s making them unreliable?
No, we know quiet a bit. The discussion moved to the esp8266 forum
http://www.esp8266.com/viewtopic.php?f=18&t=1418
and apart from agreeing that we do not know the exact reason for the crash, we suspect a timing issue and a solution (a hack) from Cal that makes it stable is available on page 11
http://www.esp8266.com/viewtopic.php?f=18&t=1418&start=100
You do need to build the fw with this small change though. Or I can upload a binary if you want.
Also, the SDK 1.0 based build that is available here
is said to not crash. Naturally this SDK has many other bug fixes. It uses more memory so I found my project does not completely fit. But I need to spend some time trimming it…
Give these a try.
[later] I should add that as we speak I have six modules (all different forms) scattered around the house reading temperature and reporting to my server, which then plots the temp and vdd of all.
This one (a nodemcu board) is hanging 75cm below the ceiling of the bedroom and runs off a 18650. You can see the battery draining then replaced with a larger one. You can see the heater kicking in at night and then cycling until morning.
One may be able to notice that I replaced the heater 10 days ago. The current one has a larger hysteresis than the old one (the first three days) and then for three nights I had one that had wide (7 dC) slow cycles that was returned. I am still tuning the new setup.
Of the six modules one is running off a wall wart (set to 3V), one off a 18650 and the rest off 3xAA (one has a 7333 LDO but the rest have no power conditioning). They are all stable now (for a couple of weeks at least).
cheers
Moving on, I want to construct a suitable power source. It seems that 3xAA will deliver over 4V initially, as will a fully charged 18650. a 3.3v regulator should be a good thing. I need an LDO with high enough driving current and low power wastage. Looking on element14 I see this as a possibility:
http://au.element14.com/texas-instruments/tps73733dcq/ic-voltage-regulator/dp/1703384
It requires little extra (one cap, and an optional second). However, as an SMD it needs a nice little board to mount it on. Something similar to this ASM7111-3.3 module:
http://www.aliexpress.com/item/-/1804418949.html
but with less components so probably smaller.
Does anyone have experience with constructing such boards?
Is this LDO a suitable one?
Yes… To start with we can just knock up an etched board.
I wouldn’t think so. It does not state what it’s quiescent current is when operating (which will be the whole time if we’re using the ESP to do timing.) The fact it’s not spec’d in the datasheet tells me it’s not going to be good.
I got 10 of these on my last element14 order:
http://au.element14.com/microchip/mcp1825s-3302e-db/ic-ldo-3-3v-500ma-sot-223-3/dp/1578404?ost=MICROCHIP++MCP1825S-3302E%2FDB
They have a typical quiescent current of 120uA, which still isn’t where I’d like it to be, but it’s not too bad. This is typically very temp dependent.
OK, looks like a good place to start. I am ready to try and put a board together (always happy to experiment on other people’s stuff). I need to learn to do this, but will later prefer to order boards instead. I hope the soldering will not be an issue, not having worked with SMD before.
My observation is that a properly working node consumes 100mAsec in 1.5 seconds of activity then sleeps (I assume 80uA) for the rest of the minute (I plan for 1m reporting but could easily accept 5m if needed). The LDO will use 120uA all the time (or only when idle? How do I interpret the Iq).
So all up per minute: 100 + (60-1.5)0.08 + 600.120 = 100 + 4.68 + 7.2 = 111.88mAsec per min = 1.86mAmin
Let’s call it 2mAh all up.
This translates to 500 hours (21 days) from a 1Ah battery. But the battery delivers 4.1-2.6V (Li-Ion) or 4.0-3.0V (3xAA). I should really work in Wh units though.
And the LDO is probably not 100% efficient - is it specified? I need to learn to read those datasheets.
All the time that it’s active. Unless we power down the ESP rail and have some other time keeping device (my idea was to use an MSP430), then we have to keep it active all the time to power the ESP.
The way linear regs work is you’re input current is approx equal to the output current + quiescent current. So efficiency isn’t really the right way of looking at it, there’s no voltage translation like there is with a switching regulator. Once you get below the target voltage + drop out voltage, then the output voltage will usually fall equal to the input voltage - dropout voltage (in this case typ. 210mV.) So you have to take this voltage drop into account when considering lowest usable battery voltage.
Powering down the whole shebang means all devices will do a cold restart. For the esp it does not matter to me (though it does have one timer that keeps running and is available across a deep sleep). The thermometer will probably take a long time to read (speced to need 750ms when otherwise I have the latest reading in 60ms). How much power does the msp430 use (I see it can go below 1uA in ultra-low-power)? And it takes this to a higher price bracket too.
I am sure TI has an msp430 base item that includes WiFi and can run the whole project (instead of the esp), right?
I don’t believe there is an MSP430 with WIFI, the msp430 is a small low power micro that wouldn’t be able to implement a full TCP/IP stack. TI have separate modules, but the price is in the order of 10x a ESP.
An MSP430 with xtal for the internal RTC will still be <$1. So personally I wouldn’t be too worried about the cost of it. It does complicate the whole thing though.
Are you using the DS18b20 sensors? There voltage range is quite a lot wider, and they have a very low standby current at <1uA.
I too would prefer to not use a second micro as well. Two sets of code makes life a lot more painful.
OK then. How about having the MSP be the orchestrator of a project, keeping time and turning things on/off. It can then be used in any project where the “slave” devices do the actual work, and they can be as complex at the task required.
The “MSP430 with xtal” still needs to be set on a board with output power rails (for the slaves), maybe logic to manage a battery (charge it too?)… looks simple but rather a big step for me (with no experience here). I wonder if people are interested in making such a thing for use in their projects (I am).
And yes, I do use the ds18b20 which needs at least 3.0v and runs on 1mA so is not too bad as long as it can be put to sleep (I currently do not actively do this but I hope it goes into standby [at 3uA] by itself when idle).
@Harvs I heard you mention the MSP430 before - how are you using it?