I have an ASP.NET Core 3.0 Web API endpoint that I have set up to allow me to post large audio files. I have followed the following directions from MS docs to set up the endpoint.
https://learn.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-3.0#kestrel-maximum-request-body-size
When an audio file is uploaded to the endpoint, it is streamed to an Azure Blob Storage container.
My code works as expected locally.
When I push it to my production server in Azure App Service on Linux, the code does not work and errors with
Unhandled exception in request pipeline: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Request body too large.
Per advice from the above article, I have configured incrementally updated Kesterl with the following:
.ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.UseKestrel((ctx, options) =>
{
var config = ctx.Configuration;
options.Limits.MaxRequestBodySize = 6000000000;
options.Limits.MinRequestBodyDataRate =
new MinDataRate(bytesPerSecond: 100,
gracePeriod: TimeSpan.FromSeconds(10));
options.Limits.MinResponseDataRate =
new MinDataRate(bytesPerSecond: 100,
gracePeriod: TimeSpan.FromSeconds(10));
options.Limits.RequestHeadersTimeout =
TimeSpan.FromMinutes(2);
}).UseStartup<Startup>();
Also configured FormOptions to accept files up to 6000000000
services.Configure<FormOptions>(options =>
{
options.MultipartBodyLengthLimit = 6000000000;
});
And also set up the API controller with the following attributes, per advice from the article
[HttpPost("audio", Name="UploadAudio")]
[DisableFormValueModelBinding]
[GenerateAntiforgeryTokenCookie]
[RequestSizeLimit(6000000000)]
[RequestFormLimits(MultipartBodyLengthLimit = 6000000000)]
Finally, here is the action itself. This giant block of code is not indicative of how I want the code to be written but I have merged it into one method as part of the debugging exercise.
public async Task<IActionResult> Audio()
{
if (!MultipartRequestHelper.IsMultipartContentType(Request.ContentType))
{
throw new ArgumentException("The media file could not be processed.");
}
string mediaId = string.Empty;
string instructorId = string.Empty;
try
{
// process file first
KeyValueAccumulator formAccumulator = new KeyValueAccumulator();
var streamedFileContent = new byte[0];
var boundary = MultipartRequestHelper.GetBoundary(
MediaTypeHeaderValue.Parse(Request.ContentType),
_defaultFormOptions.MultipartBoundaryLengthLimit
);
var reader = new MultipartReader(boundary, Request.Body);
var section = await reader.ReadNextSectionAsync();
while (section != null)
{
var hasContentDispositionHeader = ContentDispositionHeaderValue.TryParse(
section.ContentDisposition, out var contentDisposition);
if (hasContentDispositionHeader)
{
if (MultipartRequestHelper
.HasFileContentDisposition(contentDisposition))
{
streamedFileContent =
await FileHelpers.ProcessStreamedFile(section, contentDisposition,
_permittedExtensions, _fileSizeLimit);
}
else if (MultipartRequestHelper
.HasFormDataContentDisposition(contentDisposition))
{
var key = HeaderUtilities.RemoveQuotes(contentDisposition.Name).Value;
var encoding = FileHelpers.GetEncoding(section);
if (encoding == null)
{
return BadRequest($"The request could not be processed: Bad Encoding");
}
using (var streamReader = new StreamReader(
section.Body,
encoding,
detectEncodingFromByteOrderMarks: true,
bufferSize: 1024,
leaveOpen: true))
{
// The value length limit is enforced by
// MultipartBodyLengthLimit
var value = await streamReader.ReadToEndAsync();
if (string.Equals(value, "undefined",
StringComparison.OrdinalIgnoreCase))
{
value = string.Empty;
}
formAccumulator.Append(key, value);
if (formAccumulator.ValueCount >
_defaultFormOptions.ValueCountLimit)
{
return BadRequest($"The request could not be processed: Key Count limit exceeded.");
}
}
}
}
// Drain any remaining section body that hasn't been consumed and
// read the headers for the next section.
section = await reader.ReadNextSectionAsync();
}
var form = formAccumulator;
var file = streamedFileContent;
var results = form.GetResults();
instructorId = results["instructorId"];
string title = results["title"];
string firstName = results["firstName"];
string lastName = results["lastName"];
string durationInMinutes = results["durationInMinutes"];
//mediaId = await AddInstructorAudioMedia(instructorId, firstName, lastName, title, Convert.ToInt32(duration), DateTime.UtcNow, DateTime.UtcNow, file);
string fileExtension = "m4a";
// Generate Container Name - InstructorSpecific
string containerName = $"{firstName[0].ToString().ToLower()}{lastName.ToLower()}-{instructorId}";
string contentType = "audio/mp4";
FileType fileType = FileType.audio;
string authorName = $"{firstName} {lastName}";
string authorShortName = $"{firstName[0]}{lastName}";
string description = $"{authorShortName} - {title}";
long duration = (Convert.ToInt32(durationInMinutes) * 60000);
// Generate new filename
string fileName = $"{firstName[0].ToString().ToLower()}{lastName.ToLower()}-{Guid.NewGuid()}";
DateTime recordingDate = DateTime.UtcNow;
DateTime uploadDate = DateTime.UtcNow;
long blobSize = long.MinValue;
try
{
// Update file properties in storage
Dictionary<string, string> fileProperties = new Dictionary<string, string>();
fileProperties.Add("ContentType", contentType);
// update file metadata in storage
Dictionary<string, string> metadata = new Dictionary<string, string>();
metadata.Add("author", authorShortName);
metadata.Add("tite", title);
metadata.Add("description", description);
metadata.Add("duration", duration.ToString());
metadata.Add("recordingDate", recordingDate.ToString());
metadata.Add("uploadDate", uploadDate.ToString());
var fileNameWExt = $"{fileName}.{fileExtension}";
var blobContainer = await _cloudStorageService.CreateBlob(containerName, fileNameWExt, "audio");
try
{
MemoryStream fileContent = new MemoryStream(streamedFileContent);
fileContent.Position = 0;
using (fileContent)
{
await blobContainer.UploadFromStreamAsync(fileContent);
}
}
catch (StorageException e)
{
if (e.RequestInformation.HttpStatusCode == 403)
{
return BadRequest(e.Message);
}
else
{
return BadRequest(e.Message);
}
}
try
{
foreach (var key in metadata.Keys.ToList())
{
blobContainer.Metadata.Add(key, metadata[key]);
}
await blobContainer.SetMetadataAsync();
}
catch (StorageException e)
{
return BadRequest(e.Message);
}
blobSize = await StorageUtils.GetBlobSize(blobContainer);
}
catch (StorageException e)
{
return BadRequest(e.Message);
}
Media media = Media.Create(string.Empty, instructorId, authorName, fileName, fileType, fileExtension, recordingDate, uploadDate, ContentDetails.Create(title, description, duration, blobSize, 0, new List<string>()), StateDetails.Create(StatusType.STAGED, DateTime.MinValue, DateTime.UtcNow, DateTime.MaxValue), Manifest.Create(new Dictionary<string, string>()));
// upload to MongoDB
if (media != null)
{
var mapper = new Mapper(_mapperConfiguration);
var dao = mapper.Map<ContentDAO>(media);
try
{
await _db.Content.InsertOneAsync(dao);
}
catch (Exception)
{
mediaId = string.Empty;
}
mediaId = dao.Id.ToString();
}
else
{
// metadata wasn't stored, remove blob
await _cloudStorageService.DeleteBlob(containerName, fileName, "audio");
return BadRequest($"An issue occurred during media upload: rolling back storage change");
}
if (string.IsNullOrEmpty(mediaId))
{
return BadRequest($"Could not add instructor media");
}
}
catch (Exception ex)
{
return BadRequest(ex.Message);
}
var result = new { MediaId = mediaId, InstructorId = instructorId };
return Ok(result);
}
I reiterate, this all works great locally. I do not run it in IISExpress, I run it as a console app.
I submit large audio files via my SPA app and Postman and it works perfectly.
I am deploying this code to an Azure App Service on Linux (as a Basic B1).
Since the code works in my local development environment, I am at a loss of what my next steps are. I have refactored this code a few times but I suspect that it's environment related.
I cannot find anywhere that mentions that the level of App Service Plan is the culprit so before I go out spending more money I wanted to see if anyone here had encountered this challenge and could provide advice.
UPDATE: I attempted upgrading to a Production App Service Plan to see if there was an undocumented gate for incoming traffic. Upgrading didn't work either.
Thanks in advance.
-A
Currently, as of 11/2019, there is a limitation with the Azure App Service for Linux. It's CORS functionality is enabled by default and cannot be disabled AND it has a file size limitation that doesn't appear to get overridden by any of the published Kestrel configurations. The solution is to move the Web API app to a Azure App Service for Windows and it works as expected.
I am sure there is some way to get around it if you know the magic combination of configurations, server settings, and CLI commands but I need to move on with development.
When I try to load my application in Vaadin 14 I see a blank screen like this...
It appears that the servlet container is in fact running but it cannot host some static resources...
When I look in the network tab in the Chrome inspector I see it cannot find a few files such as
http://localhost:8080/VAADIN/static/client/client-122CE29AC0B9685B4DC485343E774096.cache.js
I am unsure why this would be. Any ideas?
UPDATE:
It appears I have to manually ensure all of my .jar files get added as resources in jetty so the static content can be served. I tried doing this like so and it works fine on my laptop but not on the server...
public class App {
public static void main(String[] args) throws Exception {
final var server = new Server(8080);
// Specifies the order in which the configurations are scanned.
Configuration.ClassList classlist = Configuration.ClassList.setServerDefault(server);
classlist.addAfter("org.eclipse.jetty.webapp.FragmentConfiguration", "org.eclipse.jetty.plus.webapp.EnvConfiguration", "org.eclipse.jetty.plus.webapp.PlusConfiguration");
classlist.addBefore("org.eclipse.jetty.webapp.JettyWebXmlConfiguration", "org.eclipse.jetty.annotations.AnnotationConfiguration");
// Creation of a temporal directory.
File tempDir = new File(System.getProperty("java.io.tmpdir"), "JettyTest");
if (tempDir.exists()) {
if (!tempDir.isDirectory()) {
throw new RuntimeException("Not a directory: " + tempDir);
}
} else if (!tempDir.mkdirs()) {
throw new RuntimeException("Could not make: " + tempDir);
}
WebAppContext context = new WebAppContext();
context.setInitParameter("productionMode", "false");
// Context path of the application.
context.setContextPath("");
// Exploded war or not.
context.setExtractWAR(false);
context.setTempDirectory(tempDir);
// It pulls the respective config from the VaadinServlet.
context.addServlet(GuiceVaadinServlet.class, "/*").setInitOrder(1);
context.setAttribute("org.eclipse.jetty.server.webapp.ContainerIncludeJarPattern", ".*");
context.setParentLoaderPriority(true);
server.setHandler(context);
// This add jars to the jetty classpath in a certain syntax and the pattern makes sure to load all of them.
final var classpathEntries = ClassPathHelper.getAllClassPathEntries();
final var ideMode = classpathEntries.size() > 1;
final var resourceList = new ArrayList<Resource>();
final var jarFiles = new ArrayList<File>();
if (ideMode) {
System.out.println("Starting in IDE Mode");
for (String entry : ClassPathHelper.getAllClassPathEntries()) {
if (entry.endsWith(".jar")) {
final var file = new File(entry);
jarFiles.add(file);
}
}
} else {
final var baseInstallDir = System.getProperty("user.dir");
System.out.println("Starting in Server WebJar Mode");
final var libsDirectory = new File(baseInstallDir, "lib");
System.out.println("Scanning for jars in " + libsDirectory.getPath());
for (File file : Objects.requireNonNull(libsDirectory.listFiles())) {
if (file.getPath().endsWith(".jar")) {
jarFiles.add(file);
}
}
final var sferionJar = new File(baseInstallDir, "sferion.jar");
jarFiles.add(sferionJar);
System.out.println("Found " + jarFiles.size() + " jar files");
}
for (File jarFile : jarFiles) {
resourceList.add(Resource.newResource("jar:" + jarFile.toURI().toURL() + "!/"));
}
if (ideMode) {
// It adds the web application resources. Styles, client-side components, ...
//TODO: make this property dynamic somehow?
resourceList.add(Resource.newResource("/usr/local/code/sferion/planglobal/src/main/webapp"));
}
// The base resource is where jetty serves its static content from.
context.setBaseResource(new ResourceCollection(resourceList.toArray(new Resource[0])));
server.start();
server.join();
}
}
I'm using Azure File Storage to store some files, and I want to create a zip file containing some of these files on the same Azure file share.
This is my code so far:
private void CreateZip(CloudFileDirectory directory) {
if (directory == null) throw new ArgumentNullException(nameof(directory));
var zipFilename = $"{directory.Name}.zip";
var zip = directory.GetFileReference(zipFilename);
if (!zip.Exists()) {
zip.Create(0); // <-- I don't know what size its gonna be!!
using (var zipStream = zip.OpenWrite(null))
using (var archive = new ZipArchive(zipStream, ZipArchiveMode.Create)) {
foreach (var file in directory.ListFilesAndDirectories().OfType<CloudFile>()) {
if (file.Name.Equals(zipFilename, StringComparison.InvariantCultureIgnoreCase))
continue;
using (var fileStream = file.OpenRead()) {
var entry = archive.CreateEntry(file.Name);
using (var entryStream = entry.Open())
fileStream.CopyTo(entryStream); // <-- exception is thrown
}
}
}
}
}
On the line zip.Create(0); this creates an empty file. I then go on to use this file reference to create a zip file, and add stuff to it, but when it gets to the fileStream.CopyTo(entryStream); it throws an exception with this message:
The remote server returned an error: (416) The range specified is invalid for the current size of the resource.
Presumably because the file size is 0 and it's unable to automatically increase the size.
I can create the file with int.MaxValue, but then I get a 2GB file. I can't even work out the size of the file I'm adding to the achive and resize the file to extend it by that amount, because its a zip and its gonna compress and change the file size.
How do I do this?
This issue is more related with System.IO.Compression. I have rewrite your code, please use memory stream instead like the following code. It works fine on my side. Hope it could give you some tips.
public static void CreateZip(CloudFileDirectory directory)
{
if (directory == null) throw new ArgumentNullException(nameof(directory));
var zipFilename = $"{directory.Name}.zip";
var zip = directory.GetFileReference(zipFilename);
if (!zip.Exists())
{
//zip.Create(600000); // <-- I don't know what size its gonna be!!
using (var memoryStream = new MemoryStream())
{
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true))
{
foreach (var file in directory.ListFilesAndDirectories().OfType<CloudFile>())
{
if (file.Name.Equals(zipFilename, StringComparison.InvariantCultureIgnoreCase))
continue;
using (var fileStream = file.OpenRead())
{
var entry = archive.CreateEntry(file.Name, CompressionLevel.Optimal);
using (var entryStream = entry.Open())
{
fileStream.CopyTo(entryStream); // <-- exception is thrown
}
}
}
}
memoryStream.Seek(0, SeekOrigin.Begin);
zip.UploadFromStream(memoryStream);
}
}
}
I have a requirement to run an application through my MVC controller. To get the installation path I used following link (I used answer provided by Fredrik Mörk). It worked and I could able to run the exe through a process. The problem occurred when I deployed this solution on IIS where it did not create the process as it was creating in local dev environment. Can anybody tell me how to create a windows process through a solution which is hosted on IIS ?
private string GetPathForExe(string fileName)
{
private const string keyBase = #"SOFTWARE\Wow6432Node\MyApplication";
RegistryKey localMachine = Registry.LocalMachine;
RegistryKey fileKey = localMachine.OpenSubKey(string.Format(#"{0}\{1}", keyBase, fileName));
object result = null;
if (fileKey != null)
{
result = fileKey.GetValue("InstallPath");
}
fileKey.Close();
return (string)result;
}
public void StartMyApplication()
{
Process[] pname = Process.GetProcessesByName("MyApplication");
if (pname.Length == 0)
{
string appDirectory = GetPathForExe("MyApplication");
Directory.SetCurrentDirectory(appDirectory);
ProcessStartInfo procStartInfo = new ProcessStartInfo("MyApplication.exe");
procStartInfo.WindowStyle = ProcessWindowStyle.Hidden;
Process proc = new Process();
proc.StartInfo = procStartInfo;
proc.Start();
}
}
How does one 'read' a file from a Dart program ?
http://api.dartlang.org/index.html
Dart would be running on the client-side and so taking files as input should be allowed.
You can find a usage of files in Dart's testing framework:
status_file_parser.dart (search for 'File').
In short:
File file = new File(path);
if (!file.existsSync()) <handle missing file>;
InputStream file_stream = file.openInputStream();
StringInputStream lines = new StringInputStream(file_stream);
lines.lineHandler = () {
String line;
while ((line = lines.readLine()) != null) {
...
};
lines.closeHandler = () {
...
};
Note that the API is not yet finalized and could change at any moment.
Edit: API has changed. See Introduction to new IO
Your question implies you want to do this from the client-side, that is, the browser. The dart:io library only works in the stand-alone VM on the command line.
If you do want to read a file from within the VM, there's now an easier way:
import 'dart:io';
main() {
var filename = new Options().script;
var file = new File(filename);
if (!file.existsSync()) {
print("File $filename does not exist");
return;
}
var contents = file.readAsStringSync();
print(contents);
}
If you do not want to block while the whole file is read, you can use the async version of readAsString which returns a Future:
file.readAsString().then((contents) {
print(contents);
});