I'm currently porting a pretty basic gallery application from PHP to Go. This application features automatic generation of thumbnails and middle-sized version of every image.
In PHP I used GD, because it ships with it and worked pretty well. (Code is at the end of the question). I thought I could just replicate that in Go and found go-gd from https://github.com/bolknote/go-gd (again, code is at the end). It works, but it is roughly 10 times slower (measured using time wget $URL). The PHP implementation takes about 1 second for generating a 1024x768 version from a 10 MP-image, while the Go-Code takes almost 10 seconds.
Is there any way to speed this up or any other image-processing libary for Go, which implements scaling and convolution while being reasonably fast?
PHP-Code
public function saveThumb($outName, $options) {
$this->img = imagecreatefromjpeg($filename);
if (!is_dir(dirname($outName))) {
mkdir(dirname($outName), 0777, true);
}
$width = imagesx($this->img);
$height = imagesy($this->img);
if ($options["keep_aspect"]) {
$factor = min($options["size_x"]/$width, $options["size_y"]/$height);
$new_width = round($factor*$width);
$new_height = round($factor*$height);
} else {
$new_width = $options["size_x"];
$new_height = $options["size_y"];
}
// create a new temporary image
$tmp_img = imagecreatetruecolor($new_width, $new_height);
// copy and resize old image into new image
imagecopyresampled($tmp_img, $this->img, 0, 0, 0, 0, $new_width, $new_height, $width, $height);
if ($options["sharpen"]) {
// define the sharpen matrix
$sharpen = array(
array(-1, -1.7, -1),
array(-1.7, 20, -1.7),
array(-1, -1.7, -1)
);
// calculate the sharpen divisor
$divisor = array_sum(array_map('array_sum', $sharpen));
// apply the matrix
imageconvolution($tmp_img, $sharpen, $divisor, 0);
}
// save thumbnail into a file
imagejpeg($tmp_img, $outName);
}
Go-Code
func (entry *entry) GenerateThumb(options ImageType, overwrite bool) os.Error {
targetFilename := entry.Filename(imageType)
sourceFilename := entry.Filename(IMAGE_TYPE_FULL)
targetDirname, _ := filepath.Split(targetFilename)
os.MkdirAll(targetDirname, 0777)
targetFi, errT := os.Stat(targetFilename)
sourceFi, errS := os.Stat(sourceFilename)
image := gd.CreateFromJpeg(sourceFilename)
if image == nil {
return os.NewError("Image could not be loaded")
}
var targetX, targetY int = 0, 0
if options.KeepAspect {
factor := math.Fmin(float64(options.SizeX)/float64(image.Sx()), float64(options.SizeY)/float64(image.Sy()))
targetX = int(factor*float64(image.Sx()))
targetY = int(factor*float64(image.Sy()))
} else {
targetX = options.SizeX
targetY = options.SizeY
}
tmpImage := gd.CreateTrueColor(targetX, targetY)
image.CopyResampled(tmpImage, 0, 0, 0, 0, tmpImage.Sx(), tmpImage.Sy(), image.Sx(), image.Sy())
if options.Sharpen {
sharpenMatrix := [3][3]float32{
{-1, -1.7, -1},
{-1.7, 20, -1.7},
{-1, -1.7, -1} }
tmpImage.Convolution(sharpenMatrix, 9.2, 0)
}
tmpImage.Jpeg(targetFilename, 90)
return nil
}
EDIT: Go-Code using resize.go (see answer)
func (entry *entry) GenerateThumb(options ImageType, overwrite bool) os.Error {
targetFilename := entry.Filename(imageType)
sourceFilename := entry.Filename(IMAGE_TYPE_FULL)
targetDirname, _ := filepath.Split(targetFilename)
os.MkdirAll(targetDirname, 0777)
targetFi, errT := os.Stat(targetFilename)
sourceFi, errS := os.Stat(sourceFilename)
if errT == nil && errS == nil {
if targetFi.Mtime_ns > sourceFi.Mtime_ns && !overwrite {
// already up-to-date, nothing to do
return nil
}
}
log.Printf("Generate(\"%v\", %v)\n", imageType, overwrite)
inFile, fErr := os.Open(sourceFilename)
if fErr != nil {
log.Fatal(fErr)
}
defer inFile.Close()
img, _, err := image.Decode(inFile)
if err != nil {
log.Fatal(err)
}
var targetX, targetY int
if options.KeepAspect {
factor := math.Fmin(float64(options.SizeX)/float64(img.Bounds().Max.X), float64(options.SizeY)/float64(img.Bounds().Max.Y))
targetX = int(factor*float64(img.Bounds().Max.X))
targetY = int(factor*float64(img.Bounds().Max.Y))
} else {
targetX = curType.SizeX
targetY = curType.SizeY
}
newImg := resize.Resample(img, image.Rect(0, 0, img.Bounds().Max.X, img.Bounds().Max.Y), targetX, targetY)
var outFile *os.File
outFile, fErr = os.Create(targetFilename)
if fErr != nil {
log.Fatal(fErr)
}
defer outFile.Close()
err = jpeg.Encode(outFile, newImg, &jpeg.Options{90})
if err != nil {
log.Fatal(err)
}
return nil
}
You should check out this resize library: github.com/nfnt/resize. It has 6 good interpolation functions to choose from.
The Moustachio example application for GAE by Andrew Gerrand contains a resize.go file with a native Go implementation. There was also a similar question on the go-nuts mailing list some days ago and Nigel has posted an updated version of this file there. You might want to try it :)
The easiest solution seems to save the image to disk, and execute convert from Image Magic to transform it. You can use a ram disk if you want extra performance.
Related
I'm sorting strings that are comprised of text and numbers.
I want the sort to sort the number parts as numbers, not alphanumeric.
For example I want: abc1def, ..., abc9def, abc10def
instead of: abc10def, abc1def, ..., abc9def
Does anyone know an algorithm for this (in particular in c++)
Thanks
I asked this exact question (although in Java) and got pointed to http://www.davekoelle.com/alphanum.html which has an algorithm and implementations of it in many languages.
Update 14 years later: Dave Koelle’s blog has gone off line and I can’t find his actual algorithm, but here’s an implementation.
https://github.com/cblanc/koelle-sort
Several natural sort implementations for C++ are available. A brief review:
natural_sort<> - based on Boost.Regex.
In my tests, it's roughly 20 times slower than other options.
Dirk Jagdmann's alnum.hpp, based on Dave Koelle's alphanum algorithm
Potential integer overlow issues for values over MAXINT
Martin Pool's natsort - written in C, but trivially usable from C++.
The only C/C++ implementation I've seen to offer a case insensitive version, which would seem to be a high priority for a "natural" sort.
Like the other implementations, it doesn't actually parse decimal points, but it does special case leading zeroes (anything with a leading 0 is assumed to be a fraction), which is a little weird but potentially useful.
PHP uses this algorithm.
This is known as natural sorting. There's an algorithm here that looks promising.
Be careful of problems with non-ASCII characters (see Jeff's blog entry on the subject).
Partially reposting my another answer:
bool compareNat(const std::string& a, const std::string& b){
if (a.empty())
return true;
if (b.empty())
return false;
if (std::isdigit(a[0]) && !std::isdigit(b[0]))
return true;
if (!std::isdigit(a[0]) && std::isdigit(b[0]))
return false;
if (!std::isdigit(a[0]) && !std::isdigit(b[0]))
{
if (a[0] == b[0])
return compareNat(a.substr(1), b.substr(1));
return (toUpper(a) < toUpper(b));
//toUpper() is a function to convert a std::string to uppercase.
}
// Both strings begin with digit --> parse both numbers
std::istringstream issa(a);
std::istringstream issb(b);
int ia, ib;
issa >> ia;
issb >> ib;
if (ia != ib)
return ia < ib;
// Numbers are the same --> remove numbers and recurse
std::string anew, bnew;
std::getline(issa, anew);
std::getline(issb, bnew);
return (compareNat(anew, bnew));
}
toUpper() function:
std::string toUpper(std::string s){
for(int i=0;i<(int)s.length();i++){s[i]=toupper(s[i]);}
return s;
}
Usage:
std::vector<std::string> str;
str.push_back("abc1def");
str.push_back("abc10def");
...
std::sort(str.begin(), str.end(), compareNat);
To solve what is essentially a parsing problem a state machine (aka finite state automaton) is the way to go. Dissatisfied with the above solutions i wrote a simple one-pass early bail-out algorithm that beats C/C++ variants suggested above in terms of performance, does not suffer from numerical datatype overflow errors, and is easy to modify to add case insensitivity if required.
sources can be found here
For those that arrive here and are already using Qt in their project, you can use the QCollator class. See this question for details.
Avalanchesort is a recursive variation of naturall sort, whiche merge runs, while exploring the stack of sorting-datas. The algorithim will sort stable, even if you add datas to your sorting-heap, while the algorithm is running/sorting.
The search-principle is simple. Only merge runs with the same rank.
After finding the first two naturell runs (rank 0), avalanchesort merge them to a run with rank 1. Then it call avalanchesort, to generate a second run with rank 1 and merge the two runs to a run with rank 2. Then it call the avalancheSort to generate a run with rank 2 on the unsorted datas....
My Implementation porthd/avalanchesort divide the sorting from the handling of the data using interface injection. You can use the algorithmn for datastructures like array, associative arrays or lists.
/**
* #param DataListAvalancheSortInterface $dataList
* #param DataRangeInterface $beginRange
* #param int $avalancheIndex
* #return bool
*/
public function startAvalancheSort(DataListAvalancheSortInterface $dataList)
{
$avalancheIndex = 0;
$rangeResult = $this->avalancheSort($dataList, $dataList->getFirstIdent(), $avalancheIndex);
if (!$dataList->isLastIdent($rangeResult->getStop())) {
do {
$avalancheIndex++;
$lastIdent = $rangeResult->getStop();
if ($dataList->isLastIdent($lastIdent)) {
$rangeResult = new $this->rangeClass();
$rangeResult->setStart($dataList->getFirstIdent());
$rangeResult->setStop($dataList->getLastIdent());
break;
}
$nextIdent = $dataList->getNextIdent($lastIdent);
$rangeFollow = $this->avalancheSort($dataList, $nextIdent, $avalancheIndex);
$rangeResult = $this->mergeAvalanche($dataList, $rangeResult, $rangeFollow);
} while (true);
}
return $rangeResult;
}
/**
* #param DataListAvalancheSortInterface $dataList
* #param DataRangeInterface $range
* #return DataRangeInterface
*/
protected function findRun(DataListAvalancheSortInterface $dataList,
$startIdent)
{
$result = new $this->rangeClass();
$result->setStart($startIdent);
$result->setStop($startIdent);
do {
if ($dataList->isLastIdent($result->getStop())) {
break;
}
$nextIdent = $dataList->getNextIdent($result->getStop());
if ($dataList->oddLowerEqualThanEven(
$dataList->getDataItem($result->getStop()),
$dataList->getDataItem($nextIdent)
)) {
$result->setStop($nextIdent);
} else {
break;
}
} while (true);
return $result;
}
/**
* #param DataListAvalancheSortInterface $dataList
* #param $beginIdent
* #param int $avalancheIndex
* #return DataRangeInterface|mixed
*/
protected function avalancheSort(DataListAvalancheSortInterface $dataList,
$beginIdent,
int $avalancheIndex = 0)
{
if ($avalancheIndex === 0) {
$rangeFirst = $this->findRun($dataList, $beginIdent);
if ($dataList->isLastIdent($rangeFirst->getStop())) {
// it is the last run
$rangeResult = $rangeFirst;
} else {
$nextIdent = $dataList->getNextIdent($rangeFirst->getStop());
$rangeSecond = $this->findRun($dataList, $nextIdent);
$rangeResult = $this->mergeAvalanche($dataList, $rangeFirst, $rangeSecond);
}
} else {
$rangeFirst = $this->avalancheSort($dataList,
$beginIdent,
($avalancheIndex - 1)
);
if ($dataList->isLastIdent($rangeFirst->getStop())) {
$rangeResult = $rangeFirst;
} else {
$nextIdent = $dataList->getNextIdent($rangeFirst->getStop());
$rangeSecond = $this->avalancheSort($dataList,
$nextIdent,
($avalancheIndex - 1)
);
$rangeResult = $this->mergeAvalanche($dataList, $rangeFirst, $rangeSecond);
}
}
return $rangeResult;
}
protected function mergeAvalanche(DataListAvalancheSortInterface $dataList, $oddListRange, $evenListRange)
{
$resultRange = new $this->rangeClass();
$oddNextIdent = $oddListRange->getStart();
$oddStopIdent = $oddListRange->getStop();
$evenNextIdent = $evenListRange->getStart();
$evenStopIdent = $evenListRange->getStop();
$dataList->initNewListPart($oddListRange, $evenListRange);
do {
if ($dataList->oddLowerEqualThanEven(
$dataList->getDataItem($oddNextIdent),
$dataList->getDataItem($evenNextIdent)
)) {
$dataList->addListPart($oddNextIdent);
if ($oddNextIdent === $oddStopIdent) {
$restTail = $evenNextIdent;
$stopTail = $evenStopIdent;
break;
}
$oddNextIdent = $dataList->getNextIdent($oddNextIdent);
} else {
$dataList->addListPart($evenNextIdent);
if ($evenNextIdent === $evenStopIdent) {
$restTail = $oddNextIdent;
$stopTail = $oddStopIdent;
break;
}
$evenNextIdent = $dataList->getNextIdent($evenNextIdent);
}
} while (true);
while ($stopTail !== $restTail) {
$dataList->addListPart($restTail);
$restTail = $dataList->getNextIdent($restTail);
}
$dataList->addListPart($restTail);
$dataList->cascadeDataListChange($resultRange);
return $resultRange;
}
}
My algorithm with test code of java version. If you want to use it in your project you can define a comparator yourself.
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
import java.util.function.Consumer;
public class FileNameSortTest {
private static List<String> names = Arrays.asList(
"A__01__02",
"A__2__02",
"A__1__23",
"A__11__23",
"A__3++++",
"B__1__02",
"B__22_13",
"1_22_2222",
"12_222_222",
"2222222222",
"1.sadasdsadsa",
"11.asdasdasdasdasd",
"2.sadsadasdsad",
"22.sadasdasdsadsa",
"3.asdasdsadsadsa",
"adsadsadsasd1",
"adsadsadsasd10",
"adsadsadsasd3",
"adsadsadsasd02"
);
public static void main(String...args) {
List<File> files = new ArrayList<>();
names.forEach(s -> {
File f = new File(s);
try {
if (!f.exists()) {
f.createNewFile();
}
files.add(f);
} catch (IOException e) {
e.printStackTrace();
}
});
files.sort(Comparator.comparing(File::getName));
files.forEach(f -> System.out.print(f.getName() + " "));
System.out.println();
files.sort(new Comparator<File>() {
boolean caseSensitive = false;
int SPAN_OF_CASES = 'a' - 'A';
#Override
public int compare(File left, File right) {
char[] csLeft = left.getName().toCharArray(), csRight = right.getName().toCharArray();
boolean isNumberRegion = false;
int diff=0, i=0, j=0, lenLeft=csLeft.length, lenRight=csRight.length;
char cLeft = 0, cRight = 0;
for (; i<lenLeft && j<lenRight; i++, j++) {
cLeft = getCharByCaseSensitive(csLeft[i]);
cRight = getCharByCaseSensitive(csRight[j]);
boolean isNumericLeft = isNumeric(cLeft), isNumericRight = isNumeric(cRight);
if (isNumericLeft && isNumericRight) {
// Number start!
if (!isNumberRegion) {
isNumberRegion = true;
// Remove prefix '0'
while (i < lenLeft && cLeft == '0') i++;
while (j < lenRight && cRight == '0') j++;
if (i == lenLeft || j == lenRight) break;
}
// Diff start: calculate the diff value.
if (cLeft != cRight && diff == 0)
diff = cLeft - cRight;
} else {
if (isNumericLeft != isNumericRight) {
// One numeric and one char.
if (isNumberRegion)
return isNumericLeft ? 1 : -1;
return cLeft - cRight;
} else {
// Two chars: if (number) diff don't equal 0 return it.
if (diff != 0)
return diff;
// Calculate chars diff.
diff = cLeft - cRight;
if (diff != 0)
return diff;
// Reset!
isNumberRegion = false;
diff = 0;
}
}
}
// The longer one will be put backwards.
return (i == lenLeft && j == lenRight) ? cLeft - cRight : (i == lenLeft ? -1 : 1) ;
}
private boolean isNumeric(char c) {
return c >= '0' && c <= '9';
}
private char getCharByCaseSensitive(char c) {
return caseSensitive ? c : (c >= 'A' && c <= 'Z' ? (char) (c + SPAN_OF_CASES) : c);
}
});
files.forEach(f -> System.out.print(f.getName() + " "));
}
}
The output is,
1.sadasdsadsa 11.asdasdasdasdasd 12_222_222 1_22_2222 2.sadsadasdsad 22.sadasdasdsadsa 2222222222 3.asdasdsadsadsa A__01__02 A__11__23 A__1__23 A__2__02 A__3++++ B__1__02 B__22_13 adsadsadsasd02 adsadsadsasd1 adsadsadsasd10 adsadsadsasd3
1.sadasdsadsa 1_22_2222 2.sadsadasdsad 3.asdasdsadsadsa 11.asdasdasdasdasd 12_222_222 22.sadasdasdsadsa 2222222222 A__01__02 A__1__23 A__2__02 A__3++++ A__11__23 adsadsadsasd02 adsadsadsasd1 adsadsadsasd3 adsadsadsasd10 B__1__02 B__22_13
Process finished with exit code 0
// -1: s0 < s1; 0: s0 == s1; 1: s0 > s1
static int numericCompare(const string &s0, const string &s1) {
size_t i = 0, j = 0;
for (; i < s0.size() && j < s1.size();) {
string t0(1, s0[i++]);
while (i < s0.size() && !(isdigit(t0[0]) ^ isdigit(s0[i]))) {
t0.push_back(s0[i++]);
}
string t1(1, s1[j++]);
while (j < s1.size() && !(isdigit(t1[0]) ^ isdigit(s1[j]))) {
t1.push_back(s1[j++]);
}
if (isdigit(t0[0]) && isdigit(t1[0])) {
size_t p0 = t0.find_first_not_of('0');
size_t p1 = t1.find_first_not_of('0');
t0 = p0 == string::npos ? "" : t0.substr(p0);
t1 = p1 == string::npos ? "" : t1.substr(p1);
if (t0.size() != t1.size()) {
return t0.size() < t1.size() ? -1 : 1;
}
}
if (t0 != t1) {
return t0 < t1 ? -1 : 1;
}
}
return i == s0.size() && j == s1.size() ? 0 : i != s0.size() ? 1 : -1;
}
I am not very sure if it is you want, anyway, you can have a try:-)
I captured an image from createcapture() through a webcam. Using image(), I got the image onto the canvas. Using saveframes(), and a callback, I was able to send the image to a server, but the image size is very big and I want the image to be 100 x 100 pixels every time whatever may be the size of image from saveframes() before sending it to server.
var x = window.innerWidth * 0.5;
var y = window.innerHeight * 0.75;
function setup()
{
cnv = createCanvas(x, y);
videoInput = createCapture(VIDEO);
videoInput.size(x, y);
videoInput.hide();
}
function draw()
{
if (videoInput) {
image(videoInput, 0, 0, x, y);
}
}
function image()
{
saveFrames('out','png',1,1,
function(im) {
console.log(im);
console.log(typeof im[0]);
// image Data is stored in im[0].imageData
// convert the image to smaller size
data.imageData = im[0].imageData;
// data.imageData = Math.random();
}
);
$.post('addname',data,function(result) {
console.log("data sent", result);
});
}
I have a server that sends me messages over TCP where the first 4 bytes determine the length of the rest of the message. So I need to
1) read 4 bytes into an UInt32 (works) and store it into bytes_expected
2) read bytes_expected bytes into message
Right now my code looks like this:
private let inputStreamAccessQueue = DispatchQueue(label: "SynchronizedInputStreamAccess")
func inputStreamHandler(_ event: Stream.Event) {
switch event {
case Stream.Event.hasBytesAvailable:
self.handleInput()
...
}
}
func handleInput() {
// **QUESTION: Do I use this barrier wrong?**
self.inputStreamAccessQueue.sync(flags: .barrier) {
guard let istr = self.inputStream else {
log.error(self.buildLogMessage("InputStream is nil"))
return
}
guard istr.hasBytesAvailable else {
log.error(self.buildLogMessage("handleInput() called when inputstream has no bytes available"))
return
}
let lengthbuffer = UnsafeMutablePointer<UInt8>.allocate(capacity: 4)
defer { lengthbuffer.deallocate(capacity: 4) }
let lenbytes_read = istr.read(lengthbuffer, maxLength: 4)
guard lenbytes_read == 4 else {
self.errorHandler(NetworkingError.InputError("Input Stream received \(lenbytes_read) (!=4) bytes"))
return
}
let bytes_expected = Int(UnsafeRawPointer(lengthbuffer).load(as: UInt32.self).bigEndian)
log.info(self.buildLogMessage("expect \(bytes_expected) bytes"))
print("::DEBUG", call, "bytes_expected", bytes_expected)
var message = ""
var bytes_missing = bytes_expected
while bytes_missing > 0 {
//print("::DEBUG", call, "bytes_missing", bytes_missing)
let buffer = UnsafeMutablePointer<UInt8>.allocate(capacity: bytes_missing)
let bytes_read = istr.read(buffer, maxLength: bytes_missing)
print("::DEBUG", call, "bytes_read", bytes_read)
guard bytes_read > 0 else {
print("bytes_read not > 0: \(bytes_read)")
return
}
guard bytes_read <= bytes_missing else {
print("Read more bytes than expected. missing=\(bytes_missing), read=\(bytes_read)")
return
}
guard let partial_message = String(bytesNoCopy: buffer, length: bytes_read, encoding: .utf8, freeWhenDone: true) else {
log.error("ERROR WHEN READING")
return
}
message = message + partial_message
bytes_missing -= bytes_read
}
self.handleMessage(message)
}
}
My problem is that istr.read(buffer, maxLength: bytes_missing) sometimes does not read all messages at once, so I loop until I have read all I want. But I still see my app crashing (rarely) because handleInput() is called again while another call to that method is still running. In this case, bytes_expected contains random values and the app crashes due to illegal memory allocation.
I thought I could avoid this by using the barrier. But it seems this does not work... Am I using the barrier wrong?
My suggestion is not to fight against the asynchronous nature of network I/O.
Read and collect data in a buffer whenever the Stream.Event.hasBytesAvailable event
is signalled. If the buffer contains enough data (4 length bytes plus the
expected message length) then process the data and remove it. Otherwise do nothing
and wait for more data.
The following (untested) code is meant as a demonstration.
It shows only the parts which are relevant for this particular problem.
Initialization, event handler, etc are omitted for brevity.
class MessageReader {
var buffer = Data(count: 1024) // Must be large enough for largest message + 4
var bytesRead = 0 // Number of bytes read so far
// Called from `handleInput` with a complete message.
func processMessage(message: Data) {
// ...
}
// Called from event handler if `Stream.Event.hasBytesAvailable` is signalled.
func handleInput(istr: InputStream) {
assert(bytesRead < buffer.count)
// Read from input stream, appending to previously read data:
let maxRead = buffer.count - bytesRead
let amount = buffer.withUnsafeMutableBytes { (p: UnsafeMutablePointer<UInt8>) in
istr.read(p + bytesRead, maxLength: maxRead)
}
guard amount > 0 else {
// handle EOF or read error ...
fatalError()
}
bytesRead += amount
while bytesRead >= 4 {
// Read message size:
let messageSize = buffer.withUnsafeBytes { (p: UnsafePointer<UInt32>) in
Int(UInt32(bigEndian: p.pointee))
}
let totalSize = 4 + messageSize
guard totalSize <= buffer.count else {
// Handle buffer too small for message situation ...
fatalError()
}
if bytesRead < totalSize {
break // Not enough data to read message.
}
// Buffer contains complete message now. Process it ...
processMessage(message: buffer[4 ..< totalSize])
// ... and remove it from the buffer:
if totalSize < bytesRead {
// Move remaining data to the front:
buffer.withUnsafeMutableBytes { (p: UnsafeMutablePointer<UInt8>) in
_ = memmove(p, p + totalSize, bytesRead - totalSize)
}
}
bytesRead -= totalSize
}
}
}
Inspired by Martin R (https://stackoverflow.com/a/48344040/3827381 - Thank you very much!) I came up with this solution:
var buffer = Data(count: 4096)
var offset = 0 // the index of the first byte that can be overridden
var readState = 0
var missingMsgBytes = 0
var msg = ""
func handleInput(_ istr: InputStream) {
assert(buffer.count >= 5, "buffer must be large enough to contain length info (4 bytes) and at least one payload byte => min 5 bytes buffer required")
assert(offset < buffer.count, "offset \(offset) is not smaller than \(buffer.count)")
let toRead = buffer.count - offset
let read = buffer.withUnsafeMutableBytes { (p: UnsafeMutablePointer<UInt8>) in istr.read(p + offset, maxLength: toRead) }
guard read > 0 else {
self.errorHandler(NetworkingError.InputError("Input Stream received \(read) bytes which is smaller than 0 => Network error"))
return
}
offset += read
var msgStart = 0
var msgEnd = 0
if readState == 0 {
if offset < 4 {
return
}
missingMsgBytes = buffer[0...3].withUnsafeBytes { (p: UnsafePointer<UInt32>) in Int(UInt32(bigEndian: p.pointee)) }
msgStart = 4
msgEnd = offset
readState = 1
} else {
msgStart = 0
msgEnd = offset
}
var fullMessageRead = false
if readState == 1 {
let payloadRead = msgEnd - msgStart
if payloadRead <= missingMsgBytes {
assert(msgEnd > msgStart, "msgEnd (\(msgEnd) <= msgStart \(msgStart). This should not happen")
if msgEnd > msgStart {
msg += String(data: buffer[msgStart..<msgEnd], encoding: .utf8)!
missingMsgBytes -= payloadRead
offset = 0
}
fullMessageRead = (missingMsgBytes == 0)
} else { // read more than was missing
msg += String(data: buffer[msgStart..<msgStart+missingMsgBytes], encoding: .utf8)!
fullMessageRead = true
buffer.withUnsafeMutableBytes { (p: UnsafeMutablePointer<UInt8>) in
_ = memmove(p, p + missingMsgBytes, read - missingMsgBytes) // dst, src, number
}
offset = read-missingMsgBytes
}
}
if fullMessageRead {
handleMessage(msg)
readState = 0
msg = ""
missingMsgBytes = 0
}
}
This solution is able to read messages of arbitrary size. The buffer size only determines how much can be read at one time => The bigger the buffer, the faster the app.
I tested the code for about an hour now and it did not crash. The old code crashed after 1-2 minutes. It seems to be finally working now.
But as I want to improve my programming knowledge I'd like to ask if there are some unnecessary complicated things in my code or if anyone sees a bug that could possibly still cause the app to crash or to read wrong data?
Let's consider a simple grid, where any point is connected with at most 4 other points (North-East-West-South neighborhood).
I have to write program, that computes minimal route from selected initial point to any of goal points, which are connected (there is route consisting of goal points between any two goals). Of course there can be obstacles on grid.
My solution is quite simple: I'm using A* algorithm with variable heuristic function h(x) - manhattan distance from x to nearest goal point. To find nearest goal point I have to do linear search (in O(n), where n - number of goal points). Here is my question: is there any more efficient solution (heuristic function) to dynamically find nearest goal point (where time < O(n))?
Or maybe A* is not good way to solve that problem?
How many goals, tens or thousands? If tens your way will work fine, if thousands then nearest neighbor search will give you ideas on setting up your data to search quickly.
The tradeoffs are obvious, spatially organizing your data to search will take time and on small sets brute force will be simpler to maintain. Since you're constantly evaluating I think that structuring the data will be worthwhile at very low numbers of points.
An alternate way to do this would be a modified flood fill algorithm that stops once it reaches a destination point during the flood.
First, decide whether you need to optimize, because any optimization is going to complicate your code, and for a small number of goals, your current solution is probably fine for a simple heuristic like Manhattan distance.
Before taking the first step, compute the heuristic for each goal. Remember the nearest goal as the currently selected goal, and move toward it, but subtract the maximum possible progress toward any goal from all the other distances. You can consider this second value a "meta-heuristic"; it is an optimistic estimate of the heuristic for other goals.
On subsequent steps, compute the heuristic for the current goal, and any goals with a "meta-heuristic" that is less than or equal to the heuristic. The other goals can't possibly have a better heuristic, so you don't need to compute them. The nearest goal becomes the new current goal; move toward it, subtracting the maximum possible progress from the others. Repeat until you arrive at a goal.
Use Dijkstra's algorithm, which has as it's output the minimal cost to all reachable points. Then you just select the goal points from the output.
you may consider this article If your goals not too much and want simple ways
If you want to search for any of several goals, construct a heuristic
h'(x) that is the minimum of h1(x), h2(x), h3(x), ... where h1, h2, h3
are heuristics to each of the nearby spots.
One way to think about this is that we can add a new zero-cost edge
from each of the goals to a new graph node. A path to that new node
will necessarily go through one of the goal nodes.
If you want to search for paths to all of several goals, your best
option may be Dijkstra’s Algorithm with early exit when you find all
the goals. There may be a variant of A* that can calculate these
paths; I don’t know.
If you want to search for spot near a single goal, ask A* search to
find a path to the center of the goal area. While processing nodes
from the OPEN set, exit when you pull a node that is near enough.
You can calculate the f score using the nearest target. As others said, for naive approach, you can directly calculate all target distance from current node and pick the minimum, if you only have few targets to search. For more than 100 targets, you can probably find the nearest by KDTree to speed up the process.
Here is a sample code in dart.
Iterable<Vector2> getPath(Vector2 from, Iterable<Vector2> targets,
{double? maxDistance, bool useAStar = false}) {
targets = targets.asSet();
clearPoints();
var projectedTargets = addPoints(targets).toSet();
var tree = useAStar ? IKDTree(targets) : null;
var q = PriorityQueue<Node>(_priorityQueueComparor);
Map<Vector2, Node> visited = {};
var node = Node(from);
visited[from] = node;
q.add(node);
while (q.isNotEmpty) {
var current = q.removeFirst();
// developer.log(
// '${current.point}#${current.distance}: ${getEdges(current.point).map((e) => e.dest)}');
for (var edge in getEdges(current.point)) {
if (visited.containsKey(edge.dest)) continue;
var distance = current.distance + edge.distance;
// too far
if (maxDistance != null && distance > maxDistance) continue;
// it is a target
if (projectedTargets.contains(edge.dest)) {
return reconstructPath(visited, current, edge.dest);
}
// we only interested in exploring polygon node.
if (!_polygonPoints.contains(edge.dest)) continue;
var f = 0.0;
if (tree != null) {
var nearest = tree
.nearest(edge.dest, maxDistance: maxDistance ?? double.infinity)
.firstOrNull;
f = nearest != null ? edge.dest.distanceToSquared(nearest) : 0.0;
}
node = Node(edge.dest, distance, current.count + 1, current.point, f);
visited[edge.dest] = node;
q.add(node);
}
}
return [];
}
Iterable<Vector2> reconstructPath(
Map<Vector2, Node> visited, Node prev, Vector2 point) {
var path = <Vector2>[point];
Node? currentNode = prev;
while (currentNode != null) {
path.add(currentNode.point);
currentNode = visited[currentNode.prev];
}
return path.reversed;
}
int _priorityQueueComparor(Node p0, Node p1) {
int r;
if (p0.f > 0 && p1.f > 0) {
r = ((p0.distance * p0.distance) + p0.f)
.compareTo((p1.distance * p1.distance) + p1.f);
if (r != 0) return r;
}
r = p0.distance.compareTo(p1.distance);
if (r != 0) return r;
return p0.count.compareTo(p1.count);
}
and the implementation of KDTree
class IKDTree {
final int _dimensions = 2;
late Node? _root;
IKDTree(Iterable<Vector2> points) {
_root = _buildTree(points, null);
}
Node? _buildTree(Iterable<Vector2> points, Node? parent) {
var list = points.asList();
if (list.isEmpty) return null;
var median = (list.length / 2).floor();
// Select the longest dimension as division axis
var axis = 0;
var aabb = AABB.fromPoints(list);
for (var i = 1; i < _dimensions; i++) {
if (aabb.range[i] > aabb.range[axis]) {
axis = i;
}
}
// Divide by the division axis and recursively build.
// var list = list.orderBy((e) => _selector(e)[axis]).asList();
list.sort(((a, b) => a[axis].compareTo(b[axis])));
var point = list[median];
var node = Node(point.clone());
node.parent = parent;
node.left = _buildTree(list.sublist(0, median), node);
node.right = _buildTree(list.sublist(median + 1), node);
update(node);
return node;
}
void addPoint(Vector2 point, [bool allowRebuild = true]) {
_root = _addByPoint(_root, point, allowRebuild, 0);
}
// void removePoint(Vector2 point, [bool allowRebuild = true]) {
// if (node == null) return;
// _removeNode(node, allowRebuild);
// }
Node? _addByPoint(
Node? node, Vector2 point, bool allowRebuild, int parentDim) {
if (node == null) {
node = Node(point.clone());
node.dimension = (parentDim + 1) % _dimensions;
update(node);
return node;
}
_pushDown(node);
if (point[node.dimension] < node.point[node.dimension]) {
node.left = _addByPoint(node.left, point, allowRebuild, node.dimension);
} else {
node.right = _addByPoint(node.right, point, allowRebuild, node.dimension);
}
update(node);
bool needRebuild = allowRebuild && criterionCheck(node);
if (needRebuild) node = rebuild(node);
return node;
}
// checked
void _pushDown(Node? node) {
if (node == null) return;
if (node.needPushDownToLeft && node.left != null) {
node.left!.treeDownsampleDeleted |= node.treeDownsampleDeleted;
node.left!.pointDownsampleDeleted |= node.treeDownsampleDeleted;
node.left!.treeDeleted =
node.treeDeleted || node.left!.treeDownsampleDeleted;
node.left!.deleted =
node.left!.treeDeleted || node.left!.pointDownsampleDeleted;
if (node.treeDownsampleDeleted) {
node.left!.downDeletedNum = node.left!.treeSize;
}
if (node.treeDeleted) {
node.left!.invalidNum = node.left!.treeSize;
} else {
node.left!.invalidNum = node.left!.downDeletedNum;
}
node.left!.needPushDownToLeft = true;
node.left!.needPushDownToRight = true;
node.needPushDownToLeft = false;
}
if (node.needPushDownToRight && node.right != null) {
node.right!.treeDownsampleDeleted |= node.treeDownsampleDeleted;
node.right!.pointDownsampleDeleted |= node.treeDownsampleDeleted;
node.right!.treeDeleted =
node.treeDeleted || node.right!.treeDownsampleDeleted;
node.right!.deleted =
node.right!.treeDeleted || node.right!.pointDownsampleDeleted;
if (node.treeDownsampleDeleted) {
node.right!.downDeletedNum = node.right!.treeSize;
}
if (node.treeDeleted) {
node.right!.invalidNum = node.right!.treeSize;
} else {
node.right!.invalidNum = node.right!.downDeletedNum;
}
node.right!.needPushDownToLeft = true;
node.right!.needPushDownToRight = true;
node.needPushDownToRight = false;
}
}
void _removeNode(Node? node, bool allowRebuild) {
if (node == null || node.deleted) return;
_pushDown(node);
node.deleted = true;
node.invalidNum++;
if (node.invalidNum == node.treeSize) {
node.treeDeleted = true;
}
// update and rebuild parent
var parent = node.parent;
if (parent != null) {
updateAncestors(parent);
bool needRebuild = allowRebuild && criterionCheck(parent);
if (needRebuild) parent = rebuild(parent);
}
}
void updateAncestors(Node? node) {
if (node == null) return;
update(node);
updateAncestors(node.parent);
}
void _removeByPoint(Node? node, Vector2 point, bool allowRebuild) {
if (node == null || node.treeDeleted) return;
_pushDown(node);
if (node.point == point && !node.deleted) {
node.deleted = true;
node.invalidNum++;
if (node.invalidNum == node.treeSize) {
node.treeDeleted = true;
}
return;
}
if (point[node.dimension] < node.point[node.dimension]) {
_removeByPoint(node.left, point, false);
} else {
_removeByPoint(node.right, point, false);
}
update(node);
bool needRebuild = allowRebuild && criterionCheck(node);
if (needRebuild) rebuild(node);
}
// checked
void update(Node node) {
var left = node.left;
var right = node.right;
node.treeSize = (left != null ? left.treeSize : 0) +
(right != null ? right.treeSize : 0) +
1;
node.invalidNum = (left != null ? left.invalidNum : 0) +
(right != null ? right.invalidNum : 0) +
(node.deleted ? 1 : 0);
node.downDeletedNum = (left != null ? left.downDeletedNum : 0) +
(right != null ? right.downDeletedNum : 0) +
(node.pointDownsampleDeleted ? 1 : 0);
node.treeDownsampleDeleted = (left == null || left.treeDownsampleDeleted) &&
(right == null || right.treeDownsampleDeleted) &&
node.pointDownsampleDeleted;
node.treeDeleted = (left == null || left.treeDeleted) &&
(right == null || right.treeDeleted) &&
node.deleted;
var minList = <Vector2>[];
var maxList = <Vector2>[];
if (left != null && !left.treeDeleted) {
minList.add(left.aabb.min);
maxList.add(left.aabb.max);
}
if (right != null && !right.treeDeleted) {
minList.add(right.aabb.min);
maxList.add(right.aabb.max);
}
if (!node.deleted) {
minList.add(node.point);
maxList.add(node.point);
}
if (minList.isNotEmpty && maxList.isNotEmpty) {
node.aabb = AABB()
..min = minList.min()
..max = maxList.max();
}
// TODO: Radius data for search: https://github.com/hku-mars/ikd-Tree/blob/main/ikd-Tree/ikd_Tree.cpp#L1312
if (left != null) left.parent = node;
if (right != null) right.parent = node;
// TODO: root alpha value for multithread
}
// checked
final minimalUnbalancedTreeSize = 10;
final deleteCriterionParam = 0.3;
final balanceCriterionParam = 0.6;
bool criterionCheck(Node node) {
if (node.treeSize <= minimalUnbalancedTreeSize) return false;
double balanceEvaluation = 0.0;
double deleteEvaluation = 0.0;
var child = node.left ?? node.right!;
deleteEvaluation = node.invalidNum / node.treeSize;
balanceEvaluation = child.treeSize / (node.treeSize - 1);
if (deleteEvaluation > deleteCriterionParam) return true;
if (balanceEvaluation > balanceCriterionParam ||
balanceEvaluation < 1 - balanceCriterionParam) return true;
return false;
}
void rebuildAll() {
_root = rebuild(_root);
}
// checked
Node? rebuild(Node? node) {
if (node == null) return null;
var parent = node.parent;
var points = flatten(node).toList();
// log('rebuilding: $node objects: ${objects.length}');
deleteTreeNodes(node);
return _buildTree(points, parent);
// if (parent == null) {
// _root = newNode;
// } else if (parent.left == node) {
// parent.left = newNode;
// } else if (parent.right == node) {
// parent.right = newNode;
// }
}
// checked
Iterable<Vector2> flatten(Node? node) sync* {
if (node == null) return;
_pushDown(node);
if (!node.deleted) yield node.point;
yield* flatten(node.left);
yield* flatten(node.right);
}
void deleteTreeNodes(Node? node) {
if (node == null) return;
_pushDown(node);
deleteTreeNodes(node.left);
deleteTreeNodes(node.right);
}
double _calcDist(Vector2 a, Vector2 b) {
double dist = 0;
for (var dim = 0; dim < _dimensions; dim++) {
dist += math.pow(a[dim] - b[dim], 2);
}
return dist;
}
// checked
double _calcBoxDist(Node? node, Vector2 point) {
if (node == null) return double.infinity;
double minDist = 0;
for (var dim = 0; dim < _dimensions; dim++) {
if (point[dim] < node.aabb.min[dim]) {
minDist += math.pow(point[dim] - node.aabb.min[dim], 2);
}
if (point[dim] > node.aabb.max[dim]) {
minDist += math.pow(point[dim] - node.aabb.max[dim], 2);
}
}
return minDist;
}
void _search(Node? node, int maxNodes, Vector2 point, BinaryHeap<Result> heap,
double maxDist) {
if (node == null || node.treeDeleted) return;
double curDist = _calcBoxDist(node, point);
double maxDistSqr = maxDist * maxDist;
if (curDist > maxDistSqr) return;
if (node.needPushDownToLeft || node.needPushDownToRight) {
_pushDown(node);
}
if (!node.deleted) {
double dist = _calcDist(point, node.point);
if (dist <= maxDistSqr &&
(heap.size() < maxNodes || dist < heap.peek().distance)) {
if (heap.size() >= maxNodes) heap.pop();
heap.push(Result(node, dist));
}
}
double distLeftNode = _calcBoxDist(node.left, point);
double distRightNode = _calcBoxDist(node.right, point);
if (heap.size() < maxNodes ||
distLeftNode < heap.peek().distance &&
distRightNode < heap.peek().distance) {
if (distLeftNode <= distRightNode) {
_search(node.left, maxNodes, point, heap, maxDist);
if (heap.size() < maxNodes || distRightNode < heap.peek().distance) {
_search(node.right, maxNodes, point, heap, maxDist);
}
} else {
_search(node.right, maxNodes, point, heap, maxDist);
if (heap.size() < maxNodes || distLeftNode < heap.peek().distance) {
_search(node.left, maxNodes, point, heap, maxDist);
}
}
} else {
if (distLeftNode < heap.peek().distance) {
_search(node.left, maxNodes, point, heap, maxDist);
}
if (distRightNode < heap.peek().distance) {
_search(node.right, maxNodes, point, heap, maxDist);
}
}
}
/// Find the [maxNodes] of nearest Nodes.
/// Distance is calculated via Metric function.
/// Max distance can be set with [maxDistance] param
Iterable<Vector2> nearest(Vector2 point,
{int maxNodes = 1, double maxDistance = double.infinity}) sync* {
var heap = BinaryHeap<Result>((e) => -e.distance);
_search(_root, maxNodes, point, heap, maxDistance);
var found = math.min(maxNodes, heap.content.length);
for (var i = 0; i < found; i++) {
yield heap.content[i].node.point;
}
}
int get length => _root?.length ?? 0;
int get height => _root?.height ?? 0;
}
class Result {
final Node node;
final double distance;
const Result(this.node, this.distance);
}
class Node {
Vector2 point;
int dimension = 0;
Node? parent;
Node? left;
Node? right;
int treeSize = 0;
int invalidNum = 0;
int downDeletedNum = 0;
bool deleted = false;
bool treeDeleted = false;
bool needPushDownToLeft = false;
bool needPushDownToRight = false;
bool treeDownsampleDeleted = false;
bool pointDownsampleDeleted = false;
AABB aabb = AABB();
Node(this.point);
int get length {
return 1 +
(left == null ? 0 : left!.length) +
(right == null ? 0 : right!.length);
}
int get height {
return 1 +
math.max(
left == null ? 0 : left!.height,
right == null ? 0 : right!.height,
);
}
int get depth {
return 1 + (parent == null ? 0 : parent!.depth);
}
}
I am developing a module in a billing system for a national Utility. The module is supposed to pick all successfully billed customers and print their bills.Bills are written as text files and saved on a local folder and the program has to pick them up and print them one by one.I'm using a DFX-9000 printer and pre-formatted roll paper,however,each time a new bill comes in,the printer skips some space before it prints it which distorts the 2nd and following bills.
I tried putting all the bills in a single text file which prints well when opened in notepad but not in my code.
Here is part of my code
Font printFont = new Font("Lucida Console", 10);
//static string filename;
StreamReader reader = new StreamReader(Filename);
public void Print()
{
try
{
PrintDocument pd = new PrintDocument();
pd.DefaultPageSettings.PaperSize = new System.Drawing.Printing.PaperSize("myPaper", 826, 1169);
pd.DefaultPageSettings.Margins = new Margins(0, 0, 0, 0);
//pd.DefaultPageSettings.PrinterSettings.IsPlotter = true;
pd.DefaultPageSettings.PrinterResolution.Kind = PrinterResolutionKind.Custom;
pd.PrintPage += new PrintPageEventHandler(this.PrintTextFileHandler);
pd.Print();
if (reader != null)
reader.Close();
Console.WriteLine("Printout Complete");
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
private void PrintTextFileHandler(object sender, PrintPageEventArgs pe)
{
StringFormat sf = new StringFormat();
Graphics g = pe.Graphics;
float linesPerPage = 0;
float yPos = 0;
int count = 0;
float leftMargin = 40;//pe.MarginBounds.Left;
float topMargin = pe.MarginBounds.Top;
string line = null;
linesPerPage = 500;// pe.MarginBounds.Height / printFont.GetHeight(g);
while (count <= linesPerPage &&((line = reader.ReadLine()) != null))
{
yPos = topMargin + (count * printFont.GetHeight(g));
g.DrawString(line, printFont, Brushes.Black, leftMargin, yPos);
count++;
}
if (line != null)
{
pe.HasMorePages = true;
}
else
{
pe.HasMorePages = false;
}
Could your printing.papersize be wrong? I notice it's 1169, doesn't standard paper stop at 1100?