I'm using sbcl(1.1.15) on archlinux and PortableAserve to write a web application. But I have some troubles when I using the character like "测试".
The REPL print the error only:got error The value 30334 is not of type (UNSIGNED-BYTE 8).
And the browser show nothing.
Here is my code:
(defpackage #:com.web
(:use :common-lisp :net.aserve))
(in-package :com.web)
(defun test-character-encode (req ent)
(with-http-response (req ent :content-type "text/html")
(with-http-body (req ent)
(format
(request-reply-stream req)
"测试portableallegroserve"))))
(publish :path "/test" :function 'test-character-encode)
How can I do, thanks!
The documentation for with-http-body describes an :external-format argument which may be what you need to specify:
(with-http-body (req ent &key format headers external-format) &rest body)
Within the body forms the code calls (request-reply-stream req) to
obtain a stream to which it can write to supply the body of the
response. The external-format of this stream is set to the value of
the external-format argument (which defaults to the value of
*default-aserve-external-format*). The variable *html-stream* is bound to the value of (request-reply-stream req) before the body is
evaluated. This makes it easy to use the html macro to generate
html as part of the response.
Based on this, perhaps you need something like
(with-http-body (req ent :external-format :utf-8)
(format
(request-reply-stream req)
"测试portableallegroserve"))))
Related
I have an application that sends JSON objects over AMQP, and I want to inspect the network traffic with Wireshark. The AMQP dissector gives the payload as a series of bytes in the field amqp.payload, but I'd like to extract and filter on specific fields in the JSON object, so I'm trying to write a plugin in Lua for that.
Wireshark already has a dissector for JSON, so I was hoping to piggy-back on that, and not have to deal with JSON parsing myself.
Here is my code:
local amqp_json_p = Proto("amqp_json", "AMQP JSON payload")
local amqp_json_result = ProtoField.string("amqp_json.result", "Result")
amqp_json_p.fields = { amqp_json_result }
register_postdissector(amqp_json_p)
local amqp_payload_f = Field.new("amqp.payload")
local json_dissector = Dissector.get("json")
local json_member_f = Field.new("json.member")
local json_string_f = Field.new("json.value.string")
function amqp_json_p.dissector(tvb, pinfo, tree)
local amqp_payload = amqp_payload_f()
if amqp_payload then
local payload_tvbrange = amqp_payload.range
if payload_tvbrange:range(0,1):string() == "{" then
json_dissector(payload_tvbrange:tvb(), pinfo, tree)
-- So far so good. Let's look at what the JSON dissector came up with.
local members = { json_member_f() }
local strings = { json_string_f() }
local subtree = tree:add(amqp_json_p)
for k, member in pairs(members) do
if member.display == 'result' then
for _, s in ipairs(strings) do
-- Find the string value inside this member
if not (s < member) and (s <= member) then
subtree:add(amqp_json_result, s.range)
break
end
end
end
end
end
end
end
(To start with, I'm just looking at the result field, and the payload I'm testing with is {"result":"ok"}.)
It gets me halfway there. The following shows up in the packet dissection, whereas without my plugin I only get the AMQP section:
Advanced Message Queueing Protocol
Type: Content body (3)
Channel: 1
Length: 15
Payload: 7b22726573756c74223a226f6b227d
JavaScript Object Notation
Object
Member Key: result
String value: ok
Key: result
AMQP JSON payload
Result: "ok"
Now I want to be able to use these new fields as display filters, and also to add them as columns in Wireshark. The following work for both:
json (shows up as Yes when added as a column)
json.value.string (I can also filter with json.value.string == "ok")
amqp_json
But amqp_json.result doesn't work: if I use it as a display filter, Wireshark doesn't show any packets, and if I use it as a column, the column is empty.
Why does it behave differently for json.value.string and amqp_json.result? And how can I achieve what I want? (It seems like I do need a custom dissector, as with json.value.string I can only filter on any member having a certain value, not necessarily result.)
I found a thread on the wireshark-dev mailing list ("Lua post-dissector not getting field values", 2009-09-17, 2009-09-22, 2009-09-23), that points to the interesting_hfids hash table, but it seems like the code has changed a lot since then.
If you'd like to try this, here is my PCAP file, base64-encoded, containing a single packet:
1MOyoQIABAAAAAAAAAAAAAAABAAAAAAAjBi1WfYOCgBjAAAAYwAAAB4AAABgBMEqADcGQA
AAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAB/tcWKO232y46mkSqgBgxtgA/AAAB
AQgKRjDNvkYwzb4DAAEAAAAPeyJyZXN1bHQiOiJvayJ9zg==
Decode with base64 -d (on Linux) or base64 -D (on OSX).
It turns out I shouldn't have tried to compare the display property of the json.member field. Sometimes it gets set by the JSON dissector, and sometimes it just stays as Member.
The proper solution would involve checking the value of the json.key field, but since the key I'm looking for presumably would never get escaped, I can get away with looking for the string literal in the range property of the member field.
So instead of:
if member.display == 'result' then
I have:
if member.range:range(1, 6):string() == 'result' then
and now both filtering and columns work.
Suave.Json.mapJson maps the input JSON to an object into your function, then maps the output of your function into JSON.
The problem is that I'm happy with the way it maps into my function, but I need to return a json string response rather than have suave serialise my output into JSON for me. How can I do this?
Currently i'm getting my output serialised twice. My code so far:
let executeQuery : Query -> string = //Query is my deserialised json input, the return value is a json string
let app = POST >=> path "/graphql" >=> Json.mapJson executeQuery >=> setMimeType "application/json; charset=utf-8"
startWebServer defaultConfig app
If you look at the Suave source code, you'll see that mapJson is shorthand for mapJsonWith fromJson toJson. The fromJson and toJson functions are the default JSON deserializer and serializer (respectively), but you could create your own instead -- or just use id to say "map this direction without changing it". E.g.,
let oneWayMapJson = mapJsonWith fromJson id
Note that I haven't tested this, just typed it into the Stack Overflow answer box, so some tweaking may be required. I don't have time to expand on this answer right now, but if you need more help than this rather barebones answer, let me know and I'll try to give you more help sometime tomorrow.
I would like to use non-literal strings for the "format" parameter of a logging type function, as shown here:
// You need to make c:\testDir or something similar to run this.....
//
let csvFile = #"c:\testDir\foo.csv"
open System.IO
let writer file (s:string) =
use streamWriter = new StreamWriter(file, true)
streamWriter.WriteLine(s)
// s
let log format = Printf.ksprintf (writer csvFile) format
let oneString format = (Printf.StringFormat<string->string> format)
let format = oneString "(this does not %s)"
//log format "important string"
log "this works %s" "important string"
My first attempt used a literal string, and the above fragment should work fine for you if you create the directory it needs or similar.
After discovering that you can't just "let bind" a format string, I then learned about Printf.StringFormatand more details about Printf.ksprintf, but I am obviously missing something, because I can't get them to work together with my small example.
If you comment out the last line and reinstate its predecessor, you will see a compiler error.
Making the function writer return a string almost helped (uncomment its last line), but that then makes log return a string (which means every call now needs an ignore).
I would like to know how to have my format strings dynamically settable within the type checked F# printf world!
Update
I added the parameter format to log to avoid a value restriction error that happens if log is not later called as it is in my example. I also change fmt to format in oneString.
Update
This is a different question from this one. That question does not show a function argument being passed to Printf.StringFormat (a minor difference), and it does not have the part about Printf.ksprintf not taking a continuation function that returns unit.
I thought I had found a solution with:
let oneString format = (Printf.StringFormat<string->string,unit> format)
this compiles, but there is a runtime error. (The change is the ,unit)
Rails 3:
{"a" => "<br/>"}.to_json
=> "{\"a\":\"<br/>\"}"
Rails 4:
{"a" => "<br/>"}.to_json
=> "{\"a\":\"\\u003Cbr/\\u003E\"}"
WHY???
It appears to be causing the error
Encoding::UndefinedConversionError: "\xC3" from ASCII-8BIT to UTF-8
When my Rails 3 app tries to parse JSON generated by my rails 4 app.
WHY???
To defend against a common weakness in web applications. If you say in an HTML page eg:
<script type="text/javascript">
var something = <%= #something.to_json.html_safe %>;
</script>
then you might think you're fine because you've JSON-escaped the data you're injecting into JavaScript. But actually you're not safe: aside from JSON syntax you also have surrounding HTML syntax, and in an HTML script block </ is in-band signalling. Practically, if #something contains the string </script> you've got a cross-site scripting vulnerability as this comes out:
<script type="text/javascript">
var something = {"attack": "abc</script><script>alert('XSS');//"};
</script>
The first script block ends halfway through the string (leaving an unclosed string literal syntax error) and the second <script> is treated as a new script block and the potentially-user-submitted content within it executed.
Escaping the < character to \u003C is not required by JSON but it is a perfectly valid alternative and it automatically avoids this class of problems. If a JSON parser rejects it, that is a severe bug in the reader.
What is the code that is producing that error? I'm not convinced the error is anything to do with the <-escaping, as it is talking about byte 0xC3 rather than 0x3C. That could be indicative of a string with UTF-8 encoded content not having been marked as UTF-8... maybe you need a force_encoding("UTF-8") on the input?
You can retain the original string with JSON::dump:
JSON::dump "a" => "<br/>"
=> "{\"a\":\"<br/>\"}"
JSON::dump "a" => "x&y"
=> {\"a\":\"x&y\"}" # instead of x\u0026y
Use it with care for the reasons bobince mentions and particularly avoid it with any user-generated input (or at least make sure that's sanitized).
Here's an example I encountered where it's a legitimate use. Generating a JavaScript hash argument in a helper function:
# application_helper.rb
def widget_js(post)
options = {
color: ColorCalculator(post.color).to_rgb_hex,
...
}
"third_party_widget(#{JSON::dump options});"
end
I encountered this issue too and as others have mentioned, it's caused by using the ActiveSupport to_json method. To resolve, use the JSON gem directly with JSON.generate(data) where data is an Array or Hash. See https://github.com/flori/json for all JSON gem documentation.
Was having a similar problem with Rails 7 sending "<" in JSON output like:
..., "legend":[{"text":"<96.8%","color":"#FFAFFF"},{"text":"96.8% to 98.8%","color":"#E37DE3"},{"text":"98.8% to 100%","color":"#BA50BA"}], ...
from something like:
{entry: dataset.entry, legend: dataset.legend, ...
The "<" sign was showing up "legend":[{"text":"\u003c96.8%", ...
In my case `JSON.generate({entry: ...})` fixed the issue
I have an XML as input to a Java function that parses it and produces an output. Somewhere in the XML there is the word "stratégie". The output is "stratgie". How should I parse the XML as to get the "é" character as well?
The XML is not produced by myself, I get it as a response from a web service and I am positive that "stratégie" is included in it as "stratégie".
In the parser, I have:
public List<Item> GetItems(InputStream stream) {
try {
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
DocumentBuilder db = dbf.newDocumentBuilder();
Document doc = db.parse(stream);
doc.getDocumentElement().normalize();
NodeList nodeLst = doc.getElementsByTagName("item");
List<Item> items = new ArrayList<Item>();
Item currentItem = new Item();
Node node = nodeLst.item(0);
if (node.getNodeType() == Node.ELEMENT_NODE) {
Element item = (Element) node;
if(node.getChildNodes().getLength()==0){
return null;
}
NodeList title = item.getElementsByTagName("title");
Element titleElmnt = (Element) title.item(0);
if (null != titleElmnt)
currentItem.setTitle(titleElmnt.getChildNodes().item(0).getNodeValue());
....
Using the debugger, I can see that titleElmnt.getChildNodes().item(0).getNodeValue() is "stratgie" (without the é).
Thank you for your help.
I strongly suspect that either you're parsing it incorrectly or (rather more likely) it's just not being displayed properly. You haven't really told us anything about the code or how you're using the result, which makes it hard to give very concrete advice.
As ever with encoding issues, the first thing to do is work out exactly where data is getting lost. Lots of logging tends to be the way forward: create a small test case that demonstrates the problem (as small as you can get away with) and log everything about the data. Don't just try to log it as raw text: log the Unicode value of each character. That way your log will have all the information even if there are problems with the font or encoding you use to view the log.
The answer was here: http://www.yagudaev.com/programming/java/7-jsp-escaping-html
You can either use utf-8 and have the 'é' char in your document instead of é, or you need to have a parser that understand this entity which exists in HTML and XHTML and maybe other XML dialects but not in pure XML : in pure XML there's "only" ", <, > and maybe ' I don't remember.
Maybe you can need to specify those special-char entities in your DTD or XML Schema (I don't know which one you use) and tell your parser about it.