Saxon CS: transform.doTransform cannot find out file from first transformation on windows machine but can on mac - saxon

I am creating an azure function application to validate xml files using a zip folder of schematron files.
I have run into a compatibility issue with how the URI's for the files are being created between mac and windows.
The files are downloaded from a zip on azure blob storage and then extracted to the functions local storage.
When the a colleague runs the transform method of the saxon cs api on a windows machine the method is able to run the first transformation and produce the stage 1.out file, however on the second transformation the transform method throws an exception stating that it cannot find the file even though it is present on the temp directory.
On mac the URI is /var/folders/6_/3x594vpn6z1fjclc0vx4v89m0000gn/T and on windows it is trying to find it at file:///C:/Users/44741/AppData/Local/Temp/ but the library is unable to find the file on the windows machine even if it is moved out of temp storage.
Unable to retrieve URI file:///C:/Users/44741/Desktop/files/stage1.out
The file is present at this location but for some reason the library cannot pick it up on the windows machine but it works fine on my mac. I am using Path.Combine to build the URI.
Has anyone else ran into this issue before?
The code being used for the transformations is below.
{
try
{
var transform = new Transform();
transform.doTransform(GetTransformArguments(arguments[Constants.InStage1File],
arguments[Constants.SourceDir] + "/" + schematronFile, arguments[Constants.Stage1Out]));
transform.doTransform(GetTransformArguments(arguments[Constants.InStage2File], arguments[Constants.Stage1Out],
arguments[Constants.Stage2Out]));
transform.doTransform(GetFinalTransformArguments(arguments[Constants.InStage3File], arguments[Constants.Stage2Out],
arguments[Constants.Stage3Out]));
Log.Information("Stage 3 out file written to : " + arguments[Constants.Stage3Out]);;
return true;
}
catch (FileNotFoundException ex)
{
Log.Warning("Cannot find files" + ex);
return false;
}
}
private static string[] GetTransformArguments(string xslFile, string inputFile, string outputFile)
{
return new[]
{
"-xsl:" + xslFile,
"-s:" + inputFile,
"-o:" + outputFile
};
}
private static string[] GetFinalTransformArguments(string xslFile, string inputFile, string outputFile)
{
return new[]
{
"-xsl:" + xslFile,
"-s:" + inputFile,
"-o:" + outputFile,
"allow-foreign=true",
"generate-fired-rule=true"
};
}```

So assuming the intermediary results are not needed as files but you just want the result (I assume that is the Schematron schema compiled to XSLT) you could try to run XSLT 3.0 using the API of SaxonCS (using Saxon.Api) by compiling and chaining your three stylesheets with e.g.
using Saxon.Api;
string isoSchematronDir = #"C:\SomePath\SomeDir\iso-schematron-xslt2";
string[] isoSchematronXslts = { "iso_dsdl_include.xsl", "iso_abstract_expand.xsl", "iso_svrl_for_xslt2.xsl" };
Processor processor = new(true);
var xsltCompiler = processor.NewXsltCompiler();
var baseUri = new Uri(Path.Combine(isoSchematronDir, isoSchematronXslts[2]));
xsltCompiler.BaseUri = baseUri;
var isoSchematronStages = isoSchematronXslts.Select(xslt => xsltCompiler.Compile(new Uri(baseUri, xslt)).Load30()).ToList();
isoSchematronStages[2].SetStylesheetParameters(new Dictionary<QName, XdmValue>() { { new QName("allow-foreign"), new XdmAtomicValue(true) } });
using (var schematronIs = File.OpenRead("price.sch"))
{
using (var compiledOs = File.OpenWrite("price.sch.xsl"))
{
isoSchematronStages[0].ApplyTemplates(
schematronIs,
isoSchematronStages[1].AsDocumentDestination(
isoSchematronStages[2].AsDocumentDestination(processor.NewSerializer(compiledOs)
)
);
}
}
If you only need the compiled Schematron to apply it further to validate an XML instance document against that Schematron you could even store the Schematron as an XdmDestination whose XdmNode you feed to XsltCompiler e.g.
using Saxon.Api;
string isoSchematronDir = #"C:\SomePath\SomeDir\iso-schematron-xslt2";
string[] isoSchematronXslts = { "iso_dsdl_include.xsl", "iso_abstract_expand.xsl", "iso_svrl_for_xslt2.xsl" };
Processor processor = new(true);
var xsltCompiler = processor.NewXsltCompiler();
var baseUri = new Uri(Path.Combine(isoSchematronDir, isoSchematronXslts[2]));
xsltCompiler.BaseUri = baseUri;
var isoSchematronStages = isoSchematronXslts.Select(xslt => xsltCompiler.Compile(new Uri(baseUri, xslt)).Load30()).ToList();
isoSchematronStages[2].SetStylesheetParameters(new Dictionary<QName, XdmValue>() { { new QName("allow-foreign"), new XdmAtomicValue(true) } });
var compiledSchematronXslt = new XdmDestination();
using (var schematronIs = File.OpenRead("price.sch"))
{
isoSchematronStages[0].ApplyTemplates(
schematronIs,
isoSchematronStages[1].AsDocumentDestination(
isoSchematronStages[2].AsDocumentDestination(compiledSchematronXslt)
)
);
}
var schematronValidator = xsltCompiler.Compile(compiledSchematronXslt.XdmNode).Load30();
using (var sampleIs = File.OpenRead("books.xml"))
{
schematronValidator.ApplyTemplates(sampleIs, processor.NewSerializer(Console.Out));
}
The last example writes the XSLT/Schematron validation SVRL output to the console but could of course also write it to a file.

Related

Large File upload to ASP.NET Core 3.0 Web API fails due to Request Body to Large

I have an ASP.NET Core 3.0 Web API endpoint that I have set up to allow me to post large audio files. I have followed the following directions from MS docs to set up the endpoint.
https://learn.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-3.0#kestrel-maximum-request-body-size
When an audio file is uploaded to the endpoint, it is streamed to an Azure Blob Storage container.
My code works as expected locally.
When I push it to my production server in Azure App Service on Linux, the code does not work and errors with
Unhandled exception in request pipeline: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Request body too large.
Per advice from the above article, I have configured incrementally updated Kesterl with the following:
.ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.UseKestrel((ctx, options) =>
{
var config = ctx.Configuration;
options.Limits.MaxRequestBodySize = 6000000000;
options.Limits.MinRequestBodyDataRate =
new MinDataRate(bytesPerSecond: 100,
gracePeriod: TimeSpan.FromSeconds(10));
options.Limits.MinResponseDataRate =
new MinDataRate(bytesPerSecond: 100,
gracePeriod: TimeSpan.FromSeconds(10));
options.Limits.RequestHeadersTimeout =
TimeSpan.FromMinutes(2);
}).UseStartup<Startup>();
Also configured FormOptions to accept files up to 6000000000
services.Configure<FormOptions>(options =>
{
options.MultipartBodyLengthLimit = 6000000000;
});
And also set up the API controller with the following attributes, per advice from the article
[HttpPost("audio", Name="UploadAudio")]
[DisableFormValueModelBinding]
[GenerateAntiforgeryTokenCookie]
[RequestSizeLimit(6000000000)]
[RequestFormLimits(MultipartBodyLengthLimit = 6000000000)]
Finally, here is the action itself. This giant block of code is not indicative of how I want the code to be written but I have merged it into one method as part of the debugging exercise.
public async Task<IActionResult> Audio()
{
if (!MultipartRequestHelper.IsMultipartContentType(Request.ContentType))
{
throw new ArgumentException("The media file could not be processed.");
}
string mediaId = string.Empty;
string instructorId = string.Empty;
try
{
// process file first
KeyValueAccumulator formAccumulator = new KeyValueAccumulator();
var streamedFileContent = new byte[0];
var boundary = MultipartRequestHelper.GetBoundary(
MediaTypeHeaderValue.Parse(Request.ContentType),
_defaultFormOptions.MultipartBoundaryLengthLimit
);
var reader = new MultipartReader(boundary, Request.Body);
var section = await reader.ReadNextSectionAsync();
while (section != null)
{
var hasContentDispositionHeader = ContentDispositionHeaderValue.TryParse(
section.ContentDisposition, out var contentDisposition);
if (hasContentDispositionHeader)
{
if (MultipartRequestHelper
.HasFileContentDisposition(contentDisposition))
{
streamedFileContent =
await FileHelpers.ProcessStreamedFile(section, contentDisposition,
_permittedExtensions, _fileSizeLimit);
}
else if (MultipartRequestHelper
.HasFormDataContentDisposition(contentDisposition))
{
var key = HeaderUtilities.RemoveQuotes(contentDisposition.Name).Value;
var encoding = FileHelpers.GetEncoding(section);
if (encoding == null)
{
return BadRequest($"The request could not be processed: Bad Encoding");
}
using (var streamReader = new StreamReader(
section.Body,
encoding,
detectEncodingFromByteOrderMarks: true,
bufferSize: 1024,
leaveOpen: true))
{
// The value length limit is enforced by
// MultipartBodyLengthLimit
var value = await streamReader.ReadToEndAsync();
if (string.Equals(value, "undefined",
StringComparison.OrdinalIgnoreCase))
{
value = string.Empty;
}
formAccumulator.Append(key, value);
if (formAccumulator.ValueCount >
_defaultFormOptions.ValueCountLimit)
{
return BadRequest($"The request could not be processed: Key Count limit exceeded.");
}
}
}
}
// Drain any remaining section body that hasn't been consumed and
// read the headers for the next section.
section = await reader.ReadNextSectionAsync();
}
var form = formAccumulator;
var file = streamedFileContent;
var results = form.GetResults();
instructorId = results["instructorId"];
string title = results["title"];
string firstName = results["firstName"];
string lastName = results["lastName"];
string durationInMinutes = results["durationInMinutes"];
//mediaId = await AddInstructorAudioMedia(instructorId, firstName, lastName, title, Convert.ToInt32(duration), DateTime.UtcNow, DateTime.UtcNow, file);
string fileExtension = "m4a";
// Generate Container Name - InstructorSpecific
string containerName = $"{firstName[0].ToString().ToLower()}{lastName.ToLower()}-{instructorId}";
string contentType = "audio/mp4";
FileType fileType = FileType.audio;
string authorName = $"{firstName} {lastName}";
string authorShortName = $"{firstName[0]}{lastName}";
string description = $"{authorShortName} - {title}";
long duration = (Convert.ToInt32(durationInMinutes) * 60000);
// Generate new filename
string fileName = $"{firstName[0].ToString().ToLower()}{lastName.ToLower()}-{Guid.NewGuid()}";
DateTime recordingDate = DateTime.UtcNow;
DateTime uploadDate = DateTime.UtcNow;
long blobSize = long.MinValue;
try
{
// Update file properties in storage
Dictionary<string, string> fileProperties = new Dictionary<string, string>();
fileProperties.Add("ContentType", contentType);
// update file metadata in storage
Dictionary<string, string> metadata = new Dictionary<string, string>();
metadata.Add("author", authorShortName);
metadata.Add("tite", title);
metadata.Add("description", description);
metadata.Add("duration", duration.ToString());
metadata.Add("recordingDate", recordingDate.ToString());
metadata.Add("uploadDate", uploadDate.ToString());
var fileNameWExt = $"{fileName}.{fileExtension}";
var blobContainer = await _cloudStorageService.CreateBlob(containerName, fileNameWExt, "audio");
try
{
MemoryStream fileContent = new MemoryStream(streamedFileContent);
fileContent.Position = 0;
using (fileContent)
{
await blobContainer.UploadFromStreamAsync(fileContent);
}
}
catch (StorageException e)
{
if (e.RequestInformation.HttpStatusCode == 403)
{
return BadRequest(e.Message);
}
else
{
return BadRequest(e.Message);
}
}
try
{
foreach (var key in metadata.Keys.ToList())
{
blobContainer.Metadata.Add(key, metadata[key]);
}
await blobContainer.SetMetadataAsync();
}
catch (StorageException e)
{
return BadRequest(e.Message);
}
blobSize = await StorageUtils.GetBlobSize(blobContainer);
}
catch (StorageException e)
{
return BadRequest(e.Message);
}
Media media = Media.Create(string.Empty, instructorId, authorName, fileName, fileType, fileExtension, recordingDate, uploadDate, ContentDetails.Create(title, description, duration, blobSize, 0, new List<string>()), StateDetails.Create(StatusType.STAGED, DateTime.MinValue, DateTime.UtcNow, DateTime.MaxValue), Manifest.Create(new Dictionary<string, string>()));
// upload to MongoDB
if (media != null)
{
var mapper = new Mapper(_mapperConfiguration);
var dao = mapper.Map<ContentDAO>(media);
try
{
await _db.Content.InsertOneAsync(dao);
}
catch (Exception)
{
mediaId = string.Empty;
}
mediaId = dao.Id.ToString();
}
else
{
// metadata wasn't stored, remove blob
await _cloudStorageService.DeleteBlob(containerName, fileName, "audio");
return BadRequest($"An issue occurred during media upload: rolling back storage change");
}
if (string.IsNullOrEmpty(mediaId))
{
return BadRequest($"Could not add instructor media");
}
}
catch (Exception ex)
{
return BadRequest(ex.Message);
}
var result = new { MediaId = mediaId, InstructorId = instructorId };
return Ok(result);
}
I reiterate, this all works great locally. I do not run it in IISExpress, I run it as a console app.
I submit large audio files via my SPA app and Postman and it works perfectly.
I am deploying this code to an Azure App Service on Linux (as a Basic B1).
Since the code works in my local development environment, I am at a loss of what my next steps are. I have refactored this code a few times but I suspect that it's environment related.
I cannot find anywhere that mentions that the level of App Service Plan is the culprit so before I go out spending more money I wanted to see if anyone here had encountered this challenge and could provide advice.
UPDATE: I attempted upgrading to a Production App Service Plan to see if there was an undocumented gate for incoming traffic. Upgrading didn't work either.
Thanks in advance.
-A
Currently, as of 11/2019, there is a limitation with the Azure App Service for Linux. It's CORS functionality is enabled by default and cannot be disabled AND it has a file size limitation that doesn't appear to get overridden by any of the published Kestrel configurations. The solution is to move the Web API app to a Azure App Service for Windows and it works as expected.
I am sure there is some way to get around it if you know the magic combination of configurations, server settings, and CLI commands but I need to move on with development.

Vaadin 14 app not loading cache.js files causing blank page

When I try to load my application in Vaadin 14 I see a blank screen like this...
It appears that the servlet container is in fact running but it cannot host some static resources...
When I look in the network tab in the Chrome inspector I see it cannot find a few files such as
http://localhost:8080/VAADIN/static/client/client-122CE29AC0B9685B4DC485343E774096.cache.js
I am unsure why this would be. Any ideas?
UPDATE:
It appears I have to manually ensure all of my .jar files get added as resources in jetty so the static content can be served. I tried doing this like so and it works fine on my laptop but not on the server...
public class App {
public static void main(String[] args) throws Exception {
final var server = new Server(8080);
// Specifies the order in which the configurations are scanned.
Configuration.ClassList classlist = Configuration.ClassList.setServerDefault(server);
classlist.addAfter("org.eclipse.jetty.webapp.FragmentConfiguration", "org.eclipse.jetty.plus.webapp.EnvConfiguration", "org.eclipse.jetty.plus.webapp.PlusConfiguration");
classlist.addBefore("org.eclipse.jetty.webapp.JettyWebXmlConfiguration", "org.eclipse.jetty.annotations.AnnotationConfiguration");
// Creation of a temporal directory.
File tempDir = new File(System.getProperty("java.io.tmpdir"), "JettyTest");
if (tempDir.exists()) {
if (!tempDir.isDirectory()) {
throw new RuntimeException("Not a directory: " + tempDir);
}
} else if (!tempDir.mkdirs()) {
throw new RuntimeException("Could not make: " + tempDir);
}
WebAppContext context = new WebAppContext();
context.setInitParameter("productionMode", "false");
// Context path of the application.
context.setContextPath("");
// Exploded war or not.
context.setExtractWAR(false);
context.setTempDirectory(tempDir);
// It pulls the respective config from the VaadinServlet.
context.addServlet(GuiceVaadinServlet.class, "/*").setInitOrder(1);
context.setAttribute("org.eclipse.jetty.server.webapp.ContainerIncludeJarPattern", ".*");
context.setParentLoaderPriority(true);
server.setHandler(context);
// This add jars to the jetty classpath in a certain syntax and the pattern makes sure to load all of them.
final var classpathEntries = ClassPathHelper.getAllClassPathEntries();
final var ideMode = classpathEntries.size() > 1;
final var resourceList = new ArrayList<Resource>();
final var jarFiles = new ArrayList<File>();
if (ideMode) {
System.out.println("Starting in IDE Mode");
for (String entry : ClassPathHelper.getAllClassPathEntries()) {
if (entry.endsWith(".jar")) {
final var file = new File(entry);
jarFiles.add(file);
}
}
} else {
final var baseInstallDir = System.getProperty("user.dir");
System.out.println("Starting in Server WebJar Mode");
final var libsDirectory = new File(baseInstallDir, "lib");
System.out.println("Scanning for jars in " + libsDirectory.getPath());
for (File file : Objects.requireNonNull(libsDirectory.listFiles())) {
if (file.getPath().endsWith(".jar")) {
jarFiles.add(file);
}
}
final var sferionJar = new File(baseInstallDir, "sferion.jar");
jarFiles.add(sferionJar);
System.out.println("Found " + jarFiles.size() + " jar files");
}
for (File jarFile : jarFiles) {
resourceList.add(Resource.newResource("jar:" + jarFile.toURI().toURL() + "!/"));
}
if (ideMode) {
// It adds the web application resources. Styles, client-side components, ...
//TODO: make this property dynamic somehow?
resourceList.add(Resource.newResource("/usr/local/code/sferion/planglobal/src/main/webapp"));
}
// The base resource is where jetty serves its static content from.
context.setBaseResource(new ResourceCollection(resourceList.toArray(new Resource[0])));
server.start();
server.join();
}
}

How do I increase the size of an Azure File Storage CloudFile before I know the file size?

I'm using Azure File Storage to store some files, and I want to create a zip file containing some of these files on the same Azure file share.
This is my code so far:
private void CreateZip(CloudFileDirectory directory) {
if (directory == null) throw new ArgumentNullException(nameof(directory));
var zipFilename = $"{directory.Name}.zip";
var zip = directory.GetFileReference(zipFilename);
if (!zip.Exists()) {
zip.Create(0); // <-- I don't know what size its gonna be!!
using (var zipStream = zip.OpenWrite(null))
using (var archive = new ZipArchive(zipStream, ZipArchiveMode.Create)) {
foreach (var file in directory.ListFilesAndDirectories().OfType<CloudFile>()) {
if (file.Name.Equals(zipFilename, StringComparison.InvariantCultureIgnoreCase))
continue;
using (var fileStream = file.OpenRead()) {
var entry = archive.CreateEntry(file.Name);
using (var entryStream = entry.Open())
fileStream.CopyTo(entryStream); // <-- exception is thrown
}
}
}
}
}
On the line zip.Create(0); this creates an empty file. I then go on to use this file reference to create a zip file, and add stuff to it, but when it gets to the fileStream.CopyTo(entryStream); it throws an exception with this message:
The remote server returned an error: (416) The range specified is invalid for the current size of the resource.
Presumably because the file size is 0 and it's unable to automatically increase the size.
I can create the file with int.MaxValue, but then I get a 2GB file. I can't even work out the size of the file I'm adding to the achive and resize the file to extend it by that amount, because its a zip and its gonna compress and change the file size.
How do I do this?
This issue is more related with System.IO.Compression. I have rewrite your code, please use memory stream instead like the following code. It works fine on my side. Hope it could give you some tips.
public static void CreateZip(CloudFileDirectory directory)
{
if (directory == null) throw new ArgumentNullException(nameof(directory));
var zipFilename = $"{directory.Name}.zip";
var zip = directory.GetFileReference(zipFilename);
if (!zip.Exists())
{
//zip.Create(600000); // <-- I don't know what size its gonna be!!
using (var memoryStream = new MemoryStream())
{
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true))
{
foreach (var file in directory.ListFilesAndDirectories().OfType<CloudFile>())
{
if (file.Name.Equals(zipFilename, StringComparison.InvariantCultureIgnoreCase))
continue;
using (var fileStream = file.OpenRead())
{
var entry = archive.CreateEntry(file.Name, CompressionLevel.Optimal);
using (var entryStream = entry.Open())
{
fileStream.CopyTo(entryStream); // <-- exception is thrown
}
}
}
}
memoryStream.Seek(0, SeekOrigin.Begin);
zip.UploadFromStream(memoryStream);
}
}
}

Creating a process in ASP.NET MVC controller

I have a requirement to run an application through my MVC controller. To get the installation path I used following link (I used answer provided by Fredrik Mörk). It worked and I could able to run the exe through a process. The problem occurred when I deployed this solution on IIS where it did not create the process as it was creating in local dev environment. Can anybody tell me how to create a windows process through a solution which is hosted on IIS ?
private string GetPathForExe(string fileName)
{
private const string keyBase = #"SOFTWARE\Wow6432Node\MyApplication";
RegistryKey localMachine = Registry.LocalMachine;
RegistryKey fileKey = localMachine.OpenSubKey(string.Format(#"{0}\{1}", keyBase, fileName));
object result = null;
if (fileKey != null)
{
result = fileKey.GetValue("InstallPath");
}
fileKey.Close();
return (string)result;
}
public void StartMyApplication()
{
Process[] pname = Process.GetProcessesByName("MyApplication");
if (pname.Length == 0)
{
string appDirectory = GetPathForExe("MyApplication");
Directory.SetCurrentDirectory(appDirectory);
ProcessStartInfo procStartInfo = new ProcessStartInfo("MyApplication.exe");
procStartInfo.WindowStyle = ProcessWindowStyle.Hidden;
Process proc = new Process();
proc.StartInfo = procStartInfo;
proc.Start();
}
}

How does one 'read' a file from a Dart VM program?

How does one 'read' a file from a Dart program ?
http://api.dartlang.org/index.html
Dart would be running on the client-side and so taking files as input should be allowed.
You can find a usage of files in Dart's testing framework:
status_file_parser.dart (search for 'File').
In short:
File file = new File(path);
if (!file.existsSync()) <handle missing file>;
InputStream file_stream = file.openInputStream();
StringInputStream lines = new StringInputStream(file_stream);
lines.lineHandler = () {
String line;
while ((line = lines.readLine()) != null) {
...
};
lines.closeHandler = () {
...
};
Note that the API is not yet finalized and could change at any moment.
Edit: API has changed. See Introduction to new IO
Your question implies you want to do this from the client-side, that is, the browser. The dart:io library only works in the stand-alone VM on the command line.
If you do want to read a file from within the VM, there's now an easier way:
import 'dart:io';
main() {
var filename = new Options().script;
var file = new File(filename);
if (!file.existsSync()) {
print("File $filename does not exist");
return;
}
var contents = file.readAsStringSync();
print(contents);
}
If you do not want to block while the whole file is read, you can use the async version of readAsString which returns a Future:
file.readAsString().then((contents) {
print(contents);
});

Resources