Flashing ESP8266 with a LittleFS binary - esp8266

I'm trying to build a LittleFS file system binary on my PC and flash it to my WeMos D1 Mini Pro (16MB) ESP8266.
I used the following code on the ESP
LittleFS.begin()
FSInfo info;
LittleFS.info(info);
Serial.print("LittleFS block size:");
Serial.println(info.blockSize);
Serial.print("LittleFS total bytes:");
Serial.println(info.totalBytes);
To determine the block size and total bytes, which gave me 8192 and 14655488 respectively. 14655488 / 8192 = 1789 so I used 1789 as the block size in the below python:
from littlefs import LittleFS
fs = LittleFS(block_size=8192, block_count=1789)
with open( 'index.html', 'rb' ) as f:
data = f.read()
with fs.open( '/index.html', 'w') as fh:
fh.write( data )
with open('fs.bin', 'wb') as fh:
fh.write(fs.context.buffer)
This creates a 14655488 bytes .bin file.
I then looked in boards.txt and found these lines:
d1_mini_pro.menu.eesz.16M14M=16MB (FS:14MB OTA:~1019KB)
d1_mini_pro.menu.eesz.16M14M.build.flash_size=16M
d1_mini_pro.menu.eesz.16M14M.build.flash_size_bytes=0x1000000
d1_mini_pro.menu.eesz.16M14M.build.flash_ld=eagle.flash.16m14m.ld
d1_mini_pro.menu.eesz.16M14M.build.spiffs_pagesize=256
d1_mini_pro.menu.eesz.16M14M.upload.maximum_size=1044464
d1_mini_pro.menu.eesz.16M14M.build.rfcal_addr=0xFFC000
d1_mini_pro.menu.eesz.16M14M.build.spiffs_start=0x200000
d1_mini_pro.menu.eesz.16M14M.build.spiffs_end=0xFFA000
d1_mini_pro.menu.eesz.16M14M.build.spiffs_blocksize=8192
This confirms the block size and gives the SPIFFS (but LittleFS is equivalent here, right?) start address as 0x200000
Checking these Arduino bits, i get:
FS_PHYS_ADDR: 2097152 (0x200000)
FS_PHYS_SIZE: 14655488
FS_PHYS_PAGE: 256
FS_PHYS_BLOCK: 8192
So then I used:
python upload.py --chip esp8266 --port COM6 --baud 460800 write_flash 0x200000 fs.bin
which outputs:
esptool.py v2.8
Serial port COM6
Connecting....
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: ec:fa:bc:6e:19:90
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Auto-detected Flash size: 16MB
Compressed 14655488 bytes to 215596...
Writing at 0x00234000... (100 %)
Wrote 14655488 bytes (215596 compressed) at 0x00200000 in 56.7 seconds (effective 2067.0 kbit/s)...
Hash of data verified.
Leaving...
Hard resetting via RTS pin...
However, when I then use code like the below
Dir root = LittleFS.openDir("/");
while (root.next())
{
Serial.print(root.fileName());
}
I get nothing, and
LittleFS.exists("/index.html")
returns false.
What am I doing wrong, or how do I debug this?
I'm uploading my firmware (not the filesystem) via Visual Studio Code, and the board configuration I'm using is
"xtal=80,vt=flash,exception=legacy,ssl=all,eesz=16M14M,ip=lm2f,dbg=Disabled,lvl=None____,wipe=none,baud=921600"
If I open the bin in a hex editor, I can see:
Which is some javascript inside the html file.
If I do this:
uint32_t b;
ESP.flashRead(0x006C2800 + 0x200000, &b, 1);
Serial.println(b);
then it returns 115/0x73, so it looks like the binary has flashed successfully, so that leaves me with either the binary being flashed in the wrong place, or it being corrupted/invalid....

I haven't fixed this exact problem, but I have achieved what I want.
This:
C:\Users\Andrew Bullock\AppData\Local\Arduino15\packages\esp8266\tools\mklittlefs\2.5.0-4-fe5bb56\mklittlefs.exe
exists (instructions here https://github.com/earlephilhower/mklittlefs) which works. So I can only assume that the Python wrapper or the version of LittleFS it's using is somehow incompatible or broken.

The only problem here i can see is that the block_count you gave in your python script is wrong. block_count actually refers to the pageSize of the Filesystem, that holds number of bytes every page is going to hold, in ESP littleFS the pageSize is 256.
So your object initialisation should be like this:
fs = LittleFS(block_size=8192, block_count=259)

Related

Songhe ATMega2560 + Wifi R3 ESP8266 is killing me! [WiFiEsp] >>> Cannot initialize ESP module

I am working with the Songhe Mega2560 + WiFi R3 Mega2560 + ESP8266 4MB Memory integrated circuit for a project involving connecting to a WiFi signal and reading the RSSI value.
Below is a basic sketch that I uploaded to the Mega2560 to communicate to the ESP8266 through Serial3 to test the firmware:
#include "WiFiEsp.h"
// Emulate Serial3 on pins 6/7 if not present
#ifndef HAVE_HWSERIAL3
#include "SoftwareSerial.h"
SoftwareSerial Serial3(6, 7); // RX, TX
#endif
void setup() {
// initialize serial for debugging
Serial.begin(115200);
// initialize serial for ESP module
Serial3.begin(115200);
// initialize ESP module
WiFi.init(&Serial3);
// check for the presence of the shield
if (WiFi.status() == WL_NO_SHIELD) {
Serial.println("WiFi shield not present");
// don't continue
while (true);
}
// Print WiFi MAC address
printMacAddress();
}
void loop() {
// do nothing
}
I flashed different versions of espressif's AT firmware but the serial monitor keeps showing this:
22:28:07.009 -> [WiFiEsp] Initializing ESP module
22:28:08.023 -> [WiFiEsp] >>> TIMEOUT >>>
22:28:10.026 -> [WiFiEsp] >>> TIMEOUT >>>
22:28:12.022 -> [WiFiEsp] >>> TIMEOUT >>>
22:28:14.023 -> [WiFiEsp] >>> TIMEOUT >>>
22:28:16.020 -> [WiFiEsp] >>> TIMEOUT >>>
22:28:17.006 -> [WiFiEsp] Cannot initialize ESP module
22:28:23.017 -> [WiFiEsp] >>> TIMEOUT >>>
22:28:23.017 -> [WiFiEsp] No tag found
22:28:23.017 -> WiFi shield not present
I am not sure if it is a firmware issue so I have tried multiple versions of AT firmware. The baud rate I have set is 115200. I have been looking at many other sources online, but I cannot seem to initialize WiFiEsp's WiFi module and I would really appreciate some help on this matter.
I have been following these steps for flashing and testing.
Toggle DIP switches 5,6,7 to ON and all else OFF and RXD/TXD to RXD0
Connect USB cable from port COM3 (on my computer) to integrated PCB with Mega2560 + ESP8266 WiFi
Use esptool.py to flash firmware to the ESP8266
The latest, released firmware for ESP8266 is the "ESP8266-IDF-AT_V2.2.1.0.zip" downloadable at espressif.com
I download the factory_xxx.bin to address 0 since I read that it indicates all hardware configurations for the ESP module. Below is the command I ran:
esptool.py --chip auto --port COM3 --baud 115200 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_size 4MB 0x0 factory_WROOM-02.bin
Disconnect USB cable
Toggle DIP switches 1,2,3,4 to ON and all else OFF and RXD/TXD to RXD3
Connect USB cable and upload sketch with Arduino IDE and read serial monitor
This is the procedure I have been trying to debug. If anymore information is required for better help, please let me know and I will try my best to provide.
The screenshot below is the command I run which successfully flashes (I think):
C:\Users\[MY NAME]\Downloads\ESP8266_NONOS_SDK-3.0.5\ESP8266_NONOS_SDK-3.0.5\bin>esptool.py write_flash --flash_mode dout --flash_size 4MB-c1 0x0 boot_v1.7.bin 0x01000 at/1024+1024/user1.2048.new.5.bin 0x1fb000 blank.bin 0x1fc000 esp_init_data_default_v08.bin 0xfe000 blank.bin 0x1fe000 blank.bin
esptool.py v4.4
Found 1 serial ports
Serial port COM3
Connecting....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: a4:e5:7c:b6:77:c0
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Flash will be erased from 0x00000000 to 0x00000fff...
Flash will be erased from 0x00001000 to 0x00065fff...
Flash will be erased from 0x001fb000 to 0x001fbfff...
Flash will be erased from 0x001fc000 to 0x001fcfff...
Flash will be erased from 0x000fe000 to 0x000fefff...
Flash will be erased from 0x001fe000 to 0x001fefff...
Flash params set to 0x0360
Compressed 4080 bytes to 2936...
Wrote 4080 bytes (2936 compressed) at 0x00000000 in 0.4 seconds (effective 92.5 kbit/s)...
Hash of data verified.
Compressed 413556 bytes to 296987...
Wrote 413556 bytes (296987 compressed) at 0x00001000 in 26.2 seconds (effective 126.1 kbit/s)...
Hash of data verified.
Compressed 4096 bytes to 26...
Wrote 4096 bytes (26 compressed) at 0x001fb000 in 0.1 seconds (effective 373.3 kbit/s)...
Hash of data verified.
Compressed 128 bytes to 75...
Wrote 128 bytes (75 compressed) at 0x001fc000 in 0.1 seconds (effective 11.8 kbit/s)...
Hash of data verified.
Compressed 4096 bytes to 26...
Wrote 4096 bytes (26 compressed) at 0x000fe000 in 0.1 seconds (effective 365.1 kbit/s)...
Hash of data verified.
Compressed 4096 bytes to 26...
Wrote 4096 bytes (26 compressed) at 0x001fe000 in 0.1 seconds (effective 357.1 kbit/s)...
Hash of data verified.
Leaving...
Hard resetting via RTS pin...
After this, I test it with Arduino's SerialPassThrough sketch replacing Serial1 with Serial3 and I get no response from running the command: AT.
I would appreciate any help on how to resolve this and where I could possibly be going wrong. Thanks!

pyserial issues with high baudrate FTDI

I have the following setup:
A FPGA sending out data on UART at a baudrate of 3Mbps. The data transmitted is a chunk of 1024 bytes sent at a variable periodicity ranging from 20ms to 200ms. (So even in the worst case, datarate is far from 3Msps)
A FTDI 232RG
A piece of python running on my computer (Windows), doing basically : opening a COM port with pyserial, 3Msps, polling the in_waiting until it reaches the size of a packet (1024 bytes), formatting the packet received and print it on the screen
The script works well for low repetition frequency, but I face issues with higher repetitions (typically 20ms). When the periodicity in 20ms I eventually end up getting a buffer overflow somewhere before the in_waiting. I checked the timing of my python loop and it takes about 4ms. So it looks like there is something upstream (in the FTDI or Windows) that feeds the pyserial buffer with more than one packet within the 4ms following a packet.
I tried changing the FTDI latency in the driver (from 16ms default down to a few ms) but it does not seem to help.
I am currently clueless about what is happening. Would you have any advice about how to understand better what is happening?
Thanks for your help!
You could create a "loop" between TX and RX and run the following code (tested with a FT2232H, so mostlikely you need to change the identifier string):
import time
import serial
import serial.tools.list_ports
print([(x[0],x[2]) for x in serial.tools.list_ports.comports()])
port = [x[0] for x in serial.tools.list_ports.comports() if "FT4Q1LJFB" in x[2]][0]
ser = serial.Serial(port,12000000)
while True:
t0 = time.time()
counter = 0
for i in range(1000):
ser.write([1]*3000)
recv = ser.read(ser.inWaiting())
delta_t = time.time() - t0
counter += len(recv)
print(counter / delta_t)
For me the following output is shown
[('COM7', 'USB VID:PID=0403:6010 SER=FT4Q1LJFA'), ('COM8', 'USB VID:PID=0403:6010 SER=FT4Q1LJFB')]
0.0
0.0
0.0
0.0
96787.81184093593
1201991.0268273412
1201197.0857713912
1201166.9350959768
1201445.4072856384
You will notice that it is 0.0 in the beginning. This is because I connected RX and TX after starting the program resulting in a ramping up of the received bytes. This is the "default" mode meaning 8 bits + 1 start bit + 1 stop bit = 10 bits per word which explains why "only" 1.2 Mbytes per second are transmitted.

Error in file.read() return above 2 GB on 64 bit python

I have several ~50 GB text files that I need to parse for specific contents. My files contents are organized in 4 line blocks. To perform this analysis I read in subsections of the file using file.read(chunk_size) and split into blocks of 4 then analyze them.
Because I run this script often, I've been optimizing and have tried varying the chunk size. I run 64 bit 2.7.1 python on OSX Lion on a computer with 16 GB RAM and I noticed that when I load chunks >= 2^31, instead of the expected text, I get large amounts of /x00 repeated. This continues as far as my testing has shown all the way to, and including 2^32, after which I once again get text. However, it seems that it's only returning as many characters as bytes have been added to the buffer above 4 GB.
My test code:
for i in range((2**31)-3, (2**31)+3)+range((2**32)-3, (2**32)+10):
with open('mybigtextfile.txt', 'rU') as inf:
print '%s\t%r'%(i, inf.read(i)[0:10])
My output:
2147483645 '#HWI-ST550'
2147483646 '#HWI-ST550'
2147483647 '#HWI-ST550'
2147483648 '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
2147483649 '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
2147483650 '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
4294967293 '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
4294967294 '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
4294967295 '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
4294967296 '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
4294967297 '#\x00\x00\x00\x00\x00\x00\x00\x00\x00'
4294967298 '#H\x00\x00\x00\x00\x00\x00\x00\x00'
4294967299 '#HW\x00\x00\x00\x00\x00\x00\x00'
4294967300 '#HWI\x00\x00\x00\x00\x00\x00'
4294967301 '#HWI-\x00\x00\x00\x00\x00'
4294967302 '#HWI-S\x00\x00\x00\x00'
4294967303 '#HWI-ST\x00\x00\x00'
4294967304 '#HWI-ST5\x00\x00'
4294967305 '#HWI-ST55\x00'
What exactly is going on?
Yes, this is the known issue according to the comment in cpython's source code. You can check it in Modules/_io/fileio.c. And the code add a workaround on Microsoft windows 64bit only.

Scapy - retrieving RSSI from WiFi packets

I'm trying to get RSSI or signal strength from WiFi packets.
I want also RSSI from 'WiFi probe requests' (when somebody is searching for a WiFi hotspots).
I managed to see it from kismet logs but that was only to make sure it is possible - I don't want to use kismet all the time.
For 'full time scanning' I'm using scapy. Does anybody know where can I find the RSSI or signal strength (in dBm) from the packets sniffed with scapy? I don't know how is the whole packet built - and there are a lot of 'hex' values which I don't know how to parse/interpret.
I'm sniffing on both interfaces - wlan0 (detecting when somebody connects to my hotspot), and mon.wlan0 (detecting when somebody is searching for hotspots).
Hardware (WiFi card) I use is based on Prism chipset (ISL3886). However test with Kismet was ran on Atheros (AR2413) and Intel iwl4965.
Edit1:
Looks like I need to access somehow information stored in PrismHeader:
http://trac.secdev.org/scapy/browser/scapy/layers/dot11.py
line 92 ?
Anybody knows how to enter this information?
packet.show() and packet.show2() don't show anything from this Class/Layer
Edit2:
After more digging it appears that the interface just isn't set correctly and that's why it doesn't collect all necessary headers.
If I run kismet and then sniff packets from that interface with scapy there is more info in the packet:
###[ RadioTap dummy ]###
version= 0
pad= 0
len= 26
present= TSFT+Flags+Rate+Channel+dBm_AntSignal+Antenna+b14
notdecoded= '8`/\x08\x00\x00\x00\x00\x10\x02\x94\t\xa0\x00\xdb\x01\x00\x00'
...
Now I only need to set the interface correctly without using kismet.
Here is a valuable scapy extension that improves scapy.layers.dot11.Packet's parsing of present not decoded fields.
https://github.com/ivanlei/airodump-iv/blob/master/airoiv/scapy_ex.py
Just use:
import scapy_ex
And:
packet.show()
It'll look like this:
###[ 802.11 RadioTap ]###
version = 0
pad = 0
RadioTap_len= 18
present = Flags+Rate+Channel+dBm_AntSignal+Antenna+b14
Flags = 0
Rate = 2
Channel = 1
Channel_flags= 160
dBm_AntSignal= -87
Antenna = 1
RX_Flags = 0
To summarize:
signal strength was not visible because something was wrong in the way that 'monitor mode' was set (not all headers were passed/parsed by sniffers). This monitor interface was created by hostapd.
now I'm setting monitor mode on interface with airmon-ng - tcpdump, scapy show theese extra headers.
Edited: use scapy 2.4.1+ (or github dev version). Most recent versions now correctly decode the « notdecoded » part
For some reason the packet structure has changed. Now dBm_AntSignal is the first element in notdecoded.
I am not 100% sure of this solution but I used sig_str = -(256 - ord(packet.notdecoded[-2:-1])) to reach first element and I get values that seems to be dBm_AntSignal.
I am using OpenWRT in a TP-Link MR3020 with extroot and Edward Keeble Passive Wifi Monitoring project with some modifications.
I use scapy_ex.py and I had this information:
802.11 RadioTap
version = 0
pad = 0
RadioTap_len= 36
present = dBm_AntSignal+Lock_Quality+b22+b24+b25+b26+b27+b29
dBm_AntSignal= 32
Lock_Quality= 8
If someone still has the same issue, I think I have found the solution:
I believe this is the right cut for the RSSI value:
sig_str = -(256-ord(packet.notdecoded[-3:-2]))
and this one is for the noise level:
noise_str = -(256-ord(packet.notdecoded[-2:-1]))
The fact that it says "RadioTap" suggests that the device may supply Radiotap headers, not Prism headers, even though it has a Prism chipset. The p54 driver appears to be a "SoftMAC driver", in which case it'll probably supply Radiotap headers; are you using the p54 driver or the older prism54 driver?
I have similar problem, I set up the monitor mode with airmon-ng and I can see the dBm level in tcpdump but whenever I try the sig_str = -(256-ord(packet.notdecoded[-4:-3])) I get -256 because the returned value from notdecoded in 0. Packet structure looks like this.
version = 0
pad = 0
len = 36
present = TSFT+Flags+Rate+Channel+dBm_AntSignal+b14+b29+Ext
notdecoded= ' \x08\x00\x00\x00\x00\x00\x00\x1f\x02\xed\x07\x05
.......

Detect SSD using Delphi [duplicate]

I'm getting ready to release a tool that is only effective with regular hard drives, not SSD (solid state drive). In fact, it shouldn't be used with SSD's because it will result in a lot of read/writes with no real effectiveness.
Anyone knows of a way of detecting if a given drive is solid-state?
Finally a reliable solution! Two of them, actually!
Check /sys/block/sdX/queue/rotational, where sdX is the drive name. If it's 0, you're dealing with an SSD, and 1 means plain old HDD.
I can't put my finger on the Linux version where it was introduced, but it's present in Ubuntu's Linux 3.2 and in vanilla Linux 3.6 and not present in vanilla 2.6.38. Oracle also backported it to their Unbreakable Enterprise kernel 5.5, which is based on 2.6.32.
There's also an ioctl to check if the drive is rotational since Linux 3.3, introduced by this commit. Using sysfs is usually more convenient, though.
You can actually fairly easily determine the rotational latency -- I did this once as part of a university project. It is described in this report. You'll want to skip to page 7 where you see some nice graphs of the latency. It goes from about 9.3 ms to 1.1 ms -- a drop of 8.2 ms. That corresponds directly to 60 s / 8.2 ms = 7317 RPM.
It was done with simple C code -- here's the part that measures the between positions aand b in a scratch file. We did this with larger and larger b values until we have been wandered all the way around a cylinder:
/* Measure the difference in access time between a and b. The result
* is measured in nanoseconds. */
int measure_latency(off_t a, off_t b) {
cycles_t ta, tb;
overflow_disk_buffer();
lseek(work_file, a, SEEK_SET);
read(work_file, buf, KiB/2);
ta = get_cycles();
lseek(work_file, b, SEEK_SET);
read(work_file, buf, KiB/2);
tb = get_cycles();
int diff = (tb - ta)/cycles_per_ns;
fprintf(stderr, "%i KiB to %i KiB: %i nsec\n", a / KiB, b / KiB, diff);
return diff;
}
This command lsblk -d -o name,rota lists your drives and has a 1 at ROTA if it's a rotational disk and a 0 if it's an SSD.
Example output :
NAME ROTA
sda 1
sdb 0
Detecting SSDs is not as impossible as dseifert makes out. There is already some progress in linux's libata (http://linux.derkeiler.com/Mailing-Lists/Kernel/2009-04/msg03625.html), though it doesn't seem user-ready yet.
And I definitely understand why this needs to be done. It's basically the difference between a linked list and an array. Defragmentation and such is usually counter-productive on a SSD.
You could get lucky by running
smartctl -i sda
from Smartmontools. Almost all SSDs has SSD in the Model field. No guarantee though.
My two cents to answering this old but very important question... If a disk is accessed via SCSI, then you will (potentially) be able to use SCSI INQUIRY command to request its rotational rate. VPD (Vital Product Data) page for that is called Block Device Characteristics and has a number 0xB1. Bytes 4 and 5 of this page contain a number with meaning:
0000h "Medium rotation rate is not reported"
0001h "Non-rotating medium (e.g., solid state)"
0002h - 0400h "Reserved"
0401h - FFFEh "Nominal medium rotation rate in rotations per minute (i.e.,
rpm) (e.g., 7 200 rpm = 1C20h, 10 000 rpm = 2710h, and 15 000 rpm = 3A98h)"
FFFFh "Reserved"
So, SSD must have 0001h in this field. The T10.org document about this page can be found here.
However, the implementation status of this standard is not clear to me.
I wrote the following javascript code. I needed to determine if machine was ussing SSD drive and if it was boot drive. The solution uses MSFT_PhysicalDisk WMI interface.
function main()
{
var retval= false;
// MediaType - 0 Unknown, 3 HDD, 4 SSD
// SpindleSpeed - -1 has rotational speed, 0 has no rotational speed (SSD)
// DeviceID - 0 boot device
var objWMIService = GetObject("winmgmts:\\\\.\\root\\Microsoft\\Windows\\Storage");
var colItems = objWMIService.ExecQuery("select * from MSFT_PhysicalDisk");
var enumItems = new Enumerator(colItems);
for (; !enumItems.atEnd(); enumItems.moveNext())
{
var objItem = enumItems.item();
if (objItem.MediaType == 4 && objItem.SpindleSpeed == 0)
{
if (objItem.DeviceID ==0)
{
retval=true;
}
}
}
if (retval)
{
WScript.Echo("You have SSD Drive and it is your boot drive.");
}
else
{
WScript.Echo("You do not have SSD Drive");
}
return retval;
}
main();
SSD devices emulate a hard disk device interface, so they can just be used like hard disks. This also means that there is no general way to detect what they are.
You probably could use some characteristics of the drive (latency, speed, size), though this won't be accurate for all drives. Another possibility may be to look at the S.M.A.R.T. data and see whether you can determine the type of disk through this (by model name, certain values), however unless you keep a database of all drives out there, this is not gonna be 100% accurate either.
write text file
read text file
repeat 10000 times...
10000/elapsed
for an ssd will be much higher, python3:
def ssd_test():
doc = 'ssd_test.txt'
start = time.time()
for i in range(10000):
with open(doc, 'w+') as f:
f.write('ssd test')
f.close()
with open(doc, 'r') as f:
ret = f.read()
f.close()
stop = time.time()
elapsed = stop - start
ios = int(10000/elapsed)
hd = 'HDD'
if ios > 6000: # ssd>8000; hdd <4000
hd = 'SSD'
print('detecting hard drive type by read/write speed')
print('ios', ios, 'hard drive type', hd)
return hd

Resources