Related
Using dart I made the following code that reads data from CallerId-enabled modem connected in /dev/ttyACM0 (used purely for testing - learning purpoces):
import 'package:libserialport/libserialport.dart';
import 'dart:typed_data';
void main(List<String> arguments) {
final port = SerialPort("/dev/ttyACM0");
if (!port.openReadWrite()) {
print(SerialPort.lastError);
}
try {
port.write(Uint8List.fromList("ATZ\r\n".codeUnits));
print("sent ATZ");
port.drain();
// Do not wait for response assume this is the correct command
port.write(Uint8List.fromList("AT+VCID=1\r\n".codeUnits));
port.drain();
sleep(const Duration(milliseconds: 1000));
} on SerialPortError catch (e) {
print(SerialPort.lastError);
}
final reader = SerialPortReader(port);
reader.stream.listen((data) {
print('######\nreceived: $data');
print('\nASCII:\n' + String.fromCharCodes(data) + '\nEND OF ASCII\n####');
});
}
According to modem AT command spec I expect to recieve for a phone call the following response:
RING
Whilst for Caller Id I expect a response such as:
RING
TIME=1550
DATE=10082022
NMBR=+306942870000
NAME=Chuck Norris
OK
On Both responses each newline is indicated via \r\n characters. But the output i get once I run the dart code is:
######
received: [82, 73, 78, 13, 10, 82, 73, 78, 71, 13, 10, 82, 73, 78, 71, 13, 10, 82, 73, 78, 71, 13, 10, 82, 73, 78, 71, 13, 10]
ASCII:
RIN
RING
RING
RING
RING
END OF ASCII
####
######
received: [82, 73, 78, 71, 13, 10]
ASCII:
RING
END OF ASCII
####
######
received: [82, 73, 78, 71, 13]
ASCII:
RING
END OF ASCII
####
######
received: [10]
ASCII:
END OF ASCII
####
######
received: [82, 73, 78]
ASCII:
RIN
END OF ASCII
####
######
received: [71, 13, 10]
ASCII:
G
END OF ASCII
####
As if the AT commands ATZ and AT+VCID=1 to be ignored. And ideas why that happens?
Having the same problem with you via an emperical approach, I found out that some delay is needed between opening the port and actually writing-issuing the AT commands. You can use the following piece of code in order to achieve this:
import 'package:libserialport/libserialport.dart';
import 'dart:typed_data';
import 'dart:io';
void main(List<String> arguments) {
final port = SerialPort("/dev/ttyACM0");
if (!port.openReadWrite()) {
print(SerialPort.lastError);
}
sleep(const Duration(milliseconds: 500));
try {
port.write(Uint8List.fromList("ATZ\r\n".codeUnits));
print("sent ATZ");
port.drain();
sleep(const Duration(milliseconds: 200));
// Do not wait for response assume this is the correct command
port.write(Uint8List.fromList("AT+VCID=1\r\n".codeUnits));
print("Enabled VCID");
port.drain();
sleep(const Duration(milliseconds: 1000));
} on SerialPortError catch (e) {
print(SerialPort.lastError);
}
final reader = SerialPortReader(port);
reader.stream.listen((data) {
print('######\nreceived: $data');
print('\nASCII:\n' + String.fromCharCodes(data) + '\nEND OF ASCII\n####');
});
}
I use a 500ms delay between the first write and port.openReadWrite(), in your case you can experiment with various values. That will allow to have some breathing air between writes and reads.
I'm trying to make a MOBI file parser and I'm running into a bit of an issue with trying to parse some of the binary into a struct with binary.Read().
I'm thinking it's an alignment issue, but I'm at a loss for why I'm not getting expected values. I've run the .mobi file through libmobi to test my code's output against, as well as inspected the binary of the .mobi in order to verify that I'm not crazy and the libmobi code wasn't doing something weird (which it's not).
Here's a stripped-down example:
package main
import (
"bytes"
"encoding/binary"
"fmt"
)
type Header struct {
Type [4]byte
Creator [4]byte
Uid uint32
Next uint32
RecordCount uint16
}
func main() {
testBytes := []byte{66, 79, 79, 75, 77, 79, 66, 73, 0, 0, 1, 17, 0, 0, 0, 0, 0, 136}
h := Header{}
buf := bytes.NewBuffer(testBytes)
binary.Read(buf, binary.LittleEndian, &h)
fmt.Printf("%s\n", h.Type) // BOOK, as expected
fmt.Printf("%s\n", h.Creator) // MOBI, as expected
fmt.Printf("%d\n", h.Next) // 0, as expected
fmt.Printf("%d\n", h.Uid)
// expecting Uid to be 273, but it's 285278208...
fmt.Printf("%d\n", h.RecordCount)
// expecting RecordCount to be 136, but it's 34816...
}
Any help would be greatly appreciated!
EDIT: here are the hex bytes from doing a xxd on book.mobi:
424f 4f4b 4d4f 4249 0000 0111 0000 0000 0088
Yes BigEndian works much better
package main
import (
"bytes"
"encoding/binary"
"fmt"
)
type Header struct {
Type [4]byte
Creator [4]byte
Uid uint32
Next uint32
RecordCount uint16
}
func main() {
testBytes := []byte{66, 79, 79, 75, 77, 79, 66, 73, 0, 0, 1, 17, 0, 0, 0, 0, 0, 136}
h := Header{}
buf := bytes.NewBuffer(testBytes)
binary.Read(buf, binary.BigEndian, &h)
fmt.Printf("%s\n", h.Type) // BOOK, as expected
fmt.Printf("%s\n", h.Creator) // MOBI, as expected
fmt.Printf("%d\n", h.Next) // 0, as expected
fmt.Printf("%d\n", h.Uid)
// expecting Uid to be 273, but it's 285278208...
fmt.Printf("%d\n", h.RecordCount)
// expecting RecordCount to be 136, but it's 34816...
}
connecting TCP Socket server and sending Request. and also Server sends the response in Byte array. How to read byte array data in dart.
Socket.connect('localhost', 8081)
.then((socket) {
//Establish the onData, and onDone callbacks
socket.listen((data) {
print(new String.fromCharCodes(data).trim()); //Here data is byte[]
//How to read byte array data
},
onDone: () {
print("Done");
// socket.destroy();
},
onError: (e) {
print('Server error: $e');
});
socket.add([255, 12, 0, 11, 0, 9, 34, 82, 69, 70, 84, 65, 72, 73, 76]);
});
}
It depends on with data type was encoded to bytes. Let's suppose it's String
Then you can do it with dart:convert library.
import 'dart:convert' show utf8;
final decoded = utf8.decode(data);
It's pretty clear that there's a message structure in those bytes. You give two examples of messages:
[255, 12, 0, 11, 0, 9, 34, 82, 69, 70, 84, 65, 72, 73, 76]
and
[255, 20, 0, 11, 0, 0, 0, 15, 80, 82, 69, 77, 84, 65, 72, 73, 76, 45, 53, 53, 57, 55, 48]
Both start with 255, followed by what looks like two or three little endian 16 bit words (12 and 11) and (20, 11 and 0) followed by a string, who's length is encoded in a leading byte. If you are expected to inter-operate with another system, you really need the protocol spec.
Assuming I've guessed the structure correctly, this code
main() {
Uint8List input = Uint8List.fromList([
255,
20,
0,
11,
0,
0,
0,
15,
80,
82,
69,
77,
84,
65,
72,
73,
76,
45,
53,
53,
57,
55,
48
]);
ByteData bd = input.buffer.asByteData();
print(bd.getUint16(1, Endian.little)); // print the first short
print(bd.getUint16(3, Endian.little)); // and the second
print(bd.getUint16(5, Endian.little)); // and the third
int stringLength = input[7]; // get the length of the string
print(utf8.decode(input.sublist(8, 8 + stringLength))); // decode the string
}
produces
20
11
0
PREMTAHIL-55970
as expected
I have a text file (~ 300 MB large) with a nested list, similar to this one:
[[4, 9, 11, 28, 30, 45, 55, 58, 61, 62, 63, 69, 74, 76, 77, 82, 87, 92, 93, 94, 95], [4, 9, 11, 28, 30, 45, 55, 58, 61, 62, 63, 69, 74, 76, 77, 82, 87, 92, 93, 94],[4, 9, 11, 28, 30, 45, 55, 58, 61, 62, 63, 69, 74, 76, 77, 82, 85, 87, 92, 93, 94, 95]]
Here is my program to read the file into a haskell Integer list:
import qualified Data.ByteString as ByteStr
main :: IO ()
-- HOW to do the same thing but using ByteStr.readFile for file access?
main = do fContents <- readFile filePath
let numList = readNums fContents
putStrLn (show nums)
This works for small text files, but I want to use ByteString to read the file quickly. I found out that there is no read function for ByteString, instead you should write your own parser in attoparsec, since it supports parsing ByteStrings.
How can I use attoparsec to parse the nested list?
The data seems to be in JSON format, so you can use Data.Aeson decode function which works on ByteString
import qualified Data.ByteString.Lazy as BL
import Data.Aeson
import Data.Maybe
main = do fContents <- BL.readFile filePath
let numList = decode fContents :: Maybe [[Int]]
putStrLn (show $ fromJust numList)
To export google spreadsheet's single worksheet to CSV, integer worksheet index(GID) is required to be passed.
https://spreadsheets.google.com/feeds/download/spreadsheets/Export?key=%s&gid=%d&exportFormat=csv
But, where are those informations? With gdata.spreadsheets.client, I could find some string id for worksheet like "oc6, ocv, odf".
client = gdata.spreadsheets.client.SpreadsheetsClient()
feed = client.GetWorksheets(spreadsheet, auth_token=auth_token)
And it returns below atom XML. (part of it)
<entry gd:etag=""URJFCB1NQSt7ImBoXhU."">
<id>https://spreadsheets.google.com/feeds/worksheets/0AvhN_YU3r5e9dGpTWGx3UVU3MTczaXJuNEFKQjMwN2c/ocw</id>
<updated>2012-06-21T08:19:46.587Z</updated>
<app:edited xmlns:app="http://www.w3.org/2007/app">2012-06-21T08:19:46.587Z</app:edited>
<category scheme="http://schemas.google.com/spreadsheets/2006" term="http://schemas.google.com/spreadsheets/2006#worksheet"/>
<title>AchievementType</title>
<content type="application/atom+xml;type=feed" src="https://spreadsheets.google.com/feeds/list/0AvhN_YU3r5e9dGpTWGx3UVU3MTczaXJuNEFKQjMwN2c/ocw/private/full"/>
<link rel="http://schemas.google.com/spreadsheets/2006#cellsfeed" type="application/atom+xml" href="https://spreadsheets.google.com/feeds/cells/0AvhN_YU3r5e9dGpTWGx3UVU3MTczaXJuNEFKQjMwN2c/ocw/private/full"/>
<link rel="http://schemas.google.com/visualization/2008#visualizationApi" type="application/atom+xml" href="https://spreadsheets.google.com/tq?key=0AvhN_YU3r5e9dGpTWGx3UVU3MTczaXJuNEFKQjMwN2c&sheet=ocw"/>
<link rel="self" type="application/atom+xml" href="https://spreadsheets.google.com/feeds/worksheets/0AvhN_YU3r5e9dGpTWGx3UVU3MTczaXJuNEFKQjMwN2c/private/full/ocw"/>
<link rel="edit" type="application/atom+xml" href="https://spreadsheets.google.com/feeds/worksheets/0AvhN_YU3r5e9dGpTWGx3UVU3MTczaXJuNEFKQjMwN2c/private/full/ocw"/>
<gs:rowCount>280</gs:rowCount>
<gs:colCount>28</gs:colCount>
</entry>
Also I tried with sheet parameter but failed with "Invalid Sheet" error.
https://spreadsheets.google.com/feeds/download/spreadsheets/Export?key=%s&sheet=XXX&exportFormat=csv
I guess there should be some magic function but could not find it. How can I convert them to integer id?? Or Can I export worksheet with string id?
EDIT: I just made convert table with python. DIRTY but working :-(
GID_TABLE = {
'od6': 0,
'od7': 1,
'od4': 2,
'od5': 3,
'oda': 4,
'odb': 5,
'od8': 6,
'od9': 7,
'ocy': 8,
'ocz': 9,
'ocw': 10,
'ocx': 11,
'od2': 12,
'od3': 13,
'od0': 14,
'od1': 15,
'ocq': 16,
'ocr': 17,
'oco': 18,
'ocp': 19,
'ocu': 20,
'ocv': 21,
'ocs': 22,
'oct': 23,
'oci': 24,
'ocj': 25,
'ocg': 26,
'och': 27,
'ocm': 28,
'ocn': 29,
'ock': 30,
'ocl': 31,
'oe2': 32,
'oe3': 33,
'oe0': 34,
'oe1': 35,
'oe6': 36,
'oe7': 37,
'oe4': 38,
'oe5': 39,
'odu': 40,
'odv': 41,
'ods': 42,
'odt': 43,
'ody': 44,
'odz': 45,
'odw': 46,
'odx': 47,
'odm': 48,
'odn': 49,
'odk': 50,
'odl': 51,
'odq': 52,
'odr': 53,
'odo': 54,
'odp': 55,
'ode': 56,
'odf': 57,
'odc': 58,
'odd': 59,
'odi': 60,
'odj': 61,
'odg': 62,
'odh': 63,
'obe': 64,
'obf': 65,
'obc': 66,
'obd': 67,
'obi': 68,
'obj': 69,
'obg': 70,
'obh': 71,
'ob6': 72,
'ob7': 73,
'ob4': 74,
'ob5': 75,
'oba': 76,
'obb': 77,
'ob8': 78,
'ob9': 79,
'oay': 80,
'oaz': 81,
'oaw': 82,
'oax': 83,
'ob2': 84,
'ob3': 85,
'ob0': 86,
'ob1': 87,
'oaq': 88,
'oar': 89,
'oao': 90,
'oap': 91,
'oau': 92,
'oav': 93,
'oas': 94,
'oat': 95,
'oca': 96,
'ocb': 97,
'oc8': 98,
'oc9': 99
}
I found your question looking for a solution to the same problem, and was surprised that those worksheet IDs actually correspond 1:1 to gids - I originally assumed they were assigned independently, instead of being an exercise in obfuscation.
I was able to find a slightly cleaner solution by reverse-engineering the formula they use to generate worksheet IDs from your table:
worksheetID = (gid xor 31578) encoded in base 36
So, some Python to go from a worksheet ID to gid:
def to_gid(worksheet_id):
return int(worksheet_id, 36) ^ 31578
This is still dirty, but will work for GIDs higher than 99 without requiring giant tables. At least as long as they don't change the generation logic (which they probably won't, as it would break existing IDs that people already use).
This code works with the new Google Sheets.
// Conversion of Worksheet Ids to GIDs and vice versa
// od4 > 2
function wid_to_gid(wid) {
var widval = wid.length > 3 ? wid.substring(1) : wid;
var xorval = wid.length > 3 ? 474 : 31578;
return parseInt(String(widval), 36) ^ xorval;
}
// 2 > od4
function gid_to_wid(gid) {
var xorval = gid > 31578 ? 474 : 31578;
var letter = gid > 31578 ? 'o' : '';
return letter + parseInt((gid ^ xorval)).toString(36);
}
I cannot add a comment to Wasilewski's post because apparently I lack reputation so here are the two conversion functions in Javascript based on Wasilewski's answer:
// Conversion of Worksheet Ids to GIDs and vice versa
// od4 > 2
function wid_to_gid(wid) {
return parseInt(String(wid),36)^31578
}
// 2> 0d4
function gid_to_wid(gid) {
// (gid xor 31578) encoded in base 36
return parseInt((gid^31578)).toString(36);
}
This is a Java adaptation of Buho's code which works with both the new Google Sheets and with the legacy Google Spreadsheets.
// "od4" to 2 (legacy style)
// "ogtw0h0" to 1017661118 (new style)
public static int widToGid(String worksheetId) {
boolean idIsNewStyle = worksheetId.length() > 3;
// if the id is in the new style, first strip the first character before converting
worksheetId = idIsNewStyle ? worksheetId.substring(1) : worksheetId;
// determine the integer to use for bitwise XOR
int xorValue = idIsNewStyle ? 474 : 31578;
// convert to gid
return Integer.parseInt(worksheetId, 36) ^ xorValue;
}
// Convert 2 to "od4" (legacy style)
// Convert 1017661118 to "ogtw0h0" (new style)
public static String gidToWid(int gid) {
boolean idIsNewStyle = gid > 31578;
// determine the integer to use for bitwise XOR
int xorValue = idIsNewStyle ? 474 : 31578;
// convert to worksheet id, prepending 'o' if it is the new style.
return
idIsNewStyle ?
'o' + Integer.toString((worksheetIndex ^ xorValue), 36):
Integer.toString((worksheetIndex ^ xorValue), 36);
}
This is a Clojure adaptation of Buho's and Julie's code which should work with both the new Google Sheets and with the legacy Google Spreadsheets.
(defn wid->gid [wid]
(let [new-wid? (> (.length wid) 3)
wid (if new-wid? (.substring wid 1) wid)
xor-val (if new-wid? 474 31578)]
(bit-xor (Integer/parseInt wid 36) xor-val)))
(defn gid->wid [gid]
(let [new-gid? (> gid 31578)
xor-val (if new-gid? 474 31578)
letter (if new-gid? "o" "")]
(str letter (Integer/toString (bit-xor gid xor-val) 36))))
If you're using Python with gspread, here's what you do:
wid = worksheet.id
widval = wid[1:] if len(wid) > 3 else wid
xorval = 474 if len(wid) > 3 else 31578
gid = int(str(widval), 36) ^ xorval
I'll probably open a PR for this.