I'm running Jenkins and I have it successfully working with my GitHub account, but I can't get it working correctly with Amazon S3.
I installed the S3 plugin and when I run a build it successfully uploads to the S3 bucket I specify, but all of the files uploaded end up in the root of the bucket. I have a bunch of folders (such as /css /js and so on), but all of the files in those folders from hithub end up in the root of my S3 account.
Is it possible to get the S3 plugin to upload and retain the folder structure?
It doesn't look like this is possible. Instead, I'm using s3cmd to do this. You must first install it on your server, and then in one of the bash scripts within a Jenkins job you can use:
s3cmd sync -r -P $WORKSPACE/ s3://YOUR_BUCKET_NAME
That will copy all of the files to your S3 account maintaining the folder structure. The -P keeps read permissions for everyone (needed if you're using your bucket as a web server). This is a great solution using the sync feature, because it compares all your local files against the S3 bucket and only copies files that have changed (by comparing file sizes and checksums).
I have never worked with the S3 plugin for Jenkins (but now that I know it exists, I might give it a try), though, looking at the code, it seems you can only do what you want using a workaround.
Here's what the actual plugin code does (taken from github) --I removed the parts of the code that are not relevant for the sake of readability:
class hudson.plugins.s3.S3Profile, method upload:
final Destination dest = new Destination(bucketName,filePath.getName());
getClient().putObject(dest.bucketName, dest.objectName, filePath.read(), metadata);
Now if you take a look into hudson.FilePath.getName()'s JavaDoc:
Gets just the file name portion without directories.
Now, take a look into the hudson.plugins.s3.Destination's constructor:
public Destination(final String userBucketName, final String fileName) {
if (userBucketName == null || fileName == null)
throw new IllegalArgumentException("Not defined for null parameters: "+userBucketName+","+fileName);
final String[] bucketNameArray = userBucketName.split("/", 2);
bucketName = bucketNameArray[0];
if (bucketNameArray.length > 1) {
objectName = bucketNameArray[1] + "/" + fileName;
} else {
objectName = fileName;
}
}
The Destination class JavaDoc says:
The convention implemented here is that a / in a bucket name is used to construct a structure in the object name. That is, a put of file.txt to bucket name of "mybucket/v1" will cause the object "v1/file.txt" to be created in the mybucket.
Conclusion: the filePath.getName() call strips off any prefix (S3 does not have any directory, but rather prefixes, see this and this threads for more info) you add to the file. If you really need to put your files into a "folder" (i.e. having a specific prefix that contains a slash (/)), I suggest you to add this prefix to the end of your bucket name, as explicited in the Destination class JavaDoc.
Yes this is possible.
It looks like for each folder destination, you'll need a separate instance of the S3 plugin however.
"Source" is the file you're uploading.
"Destination bucket" is where you place your path.
Using Jenkins 1.532.2 and S3 Publisher Plug-In 0.5, the UI configure Job screen rejects additional S3 publish entries. There would also be a significant maintenance benefit to us if the plugin recreated the workspace directory structure as we'll have many directories to create.
Set up your git plugin.
Set up your Bash script
All in your folder marked as "*" will go to bucket
Related
I need to upload a zip file to an s3 bucket after its creation. I'm aware of the s3_deployment package but it doesn't fit my usecase because I need the file to be uploaded only once, on stack creation. The s3_deployment package would upload the zip on every update.
I have the following custom resource defined however I'm not sure how to pass the body of the file to the custom resource. I've tried opening the file in binary mode but that returns an error.
app_data_bootstrap = AwsCustomResource(self, "BootstrapData",
on_create={
"service": "S3",
"action": "putObject",
"parameters": {
"Body": open('app_data.zip', 'rb'),
"Bucket": f"my-app-data",
"Key": "app_data.zip",
},
"physical_resource_id": PhysicalResourceId.of("BootstrapDataBucket")
},
policy=AwsCustomResourcePolicy.from_sdk_calls(resources=AwsCustomResourcePolicy.ANY_RESOURCE)
)
I don't believe that's possible unless you write a custom script and runs before your cdk deploy to upload your local files to an intermediary S3 bucket. Then you can write a custom resource that copies content of the intermediary bucket on on_create event to the bucket that was created via CDK.
Read this paragraph from s3_deployment in CDK docs:
This is what happens under the hood:
When this stack is deployed (either via cdk deploy or via CI/CD), the contents of the local website-dist directory will be archived and uploaded to an intermediary assets bucket. If there is more than one source, they will be individually uploaded.
The BucketDeployment construct synthesizes a custom CloudFormation resource of type Custom::CDKBucketDeployment into the template. The source bucket/key is set to point to the assets bucket.
The custom resource downloads the .zip archive, extracts it and issues aws s3 sync --delete against the destination bucket (in this case websiteBucket). If there is more than one source, the sources will be downloaded and merged pre-deployment at this step.
So in order for you do replicate step 1, you have to write a small script that creates an intermediate bucket and uploads your local files to it. A sample of that script can be like this:
#!/bin/sh
aws s3 mb <intermediary_bucket> --region <region_name>
aws s3 sync <intermediary_bucket> s3://<your_bucket_name>
Then your custom resource can be something like this:
*Note that this will work for copying one object, you can change the code to copy multiple objects.
import json
import boto3
import cfnresponse
def lambda_handler(event, context):
print('Received request:\n%s' % json.dumps(event, indent=4))
resource_properties = event['ResourceProperties']
if event['RequestType'] in ['Create']: #What happens when resource is created
try:
s3 = boto3.resource('s3')
copy_source = {
'Bucket': 'intermediary_bucket',
'Key': 'path/to/filename.extension'
}
bucket = s3.Bucket('otherbucket')
obj = bucket.Object('otherkey')
obj.copy(copy_source)
except:
cfnresponse.send(event, context, cfnresponse.FAILED, {})
raise
else:
cfnresponse.send(event, context, cfnresponse.SUCCESS,
{'FileContent': response['fileContent'].decode('utf-8')})
elif event['RequestType'] == 'Delete': # What happens when resource is deleted
cfnresponse.send(event, context, cfnresponse.SUCCESS, {})
Alternative to all of this, is to open an issue in AWS CDK's Github repo and ask them to add your usecase.
I'm using Grails 2.5.6 to store uploaded images to folder on a server.
The following are my code to store the image
mpr.multiFileMap.file.each{fileData->
CommonsMultipartFile file = fileData
File convFile = new File(file.getOriginalFilename());
file.transferTo(convFile);
/** Processing File **/
File uploadedFile = new File("${directory}${generatedFileName}.${extension}")
convFile.renameTo(uploadedFile)
}
I have no problem running on development (MacOSX High Sierra)
But when i deployed on production (Ubuntu 14.04 server), i could not save the file outside CATALINA_HOME directory.
I have checked the permission and ownership of the destination directory, but still, the directory was created but the file was never stored.
For Example, i've tried to store the file on /home/tomcat/ directory (/home directory was in separate partition with tomcat which stored it /var), the directory was created, but the file was never stored.
When i put the destination directory within CATALINA_HOME folder, everything works fine. But this was not the scenario i want to do.
You say your destination directory is on another partition, so maybe another filesystem is used on this partition.
Or if you look on the javadoc of the renameTo method it is said :
Many aspects of the behavior of this method are inherently
platform-dependent: The rename operation might not be able to move a
file from one filesystem to another, it might not be atomic, and it
might not succeed if a file with the destination abstract pathname
already exists. The return value should always be checked to make
sure that the rename operation was successful.
...
#return true if and only if the renaming succeeded;
false otherwise
Thus I think the renameTo method is not able to move the file, don't know why but you can rewrite your code like this :
mpr.multiFileMap.file.each{fileData->
CommonsMultipartFile file = fileData
File uploadedFile = new File("${directory}${generatedFileName}.${extension}")
// String originalFilename = file.getOriginalFilename()
// you can store originalFilename in database for example
if(!uploadedFile.getParentFile().exists()) {
uploadedFile.getParentFile().mkdirs()
// You can set permissions on the target directory if you desired, using PosixFilePermission
}
file.transferTo(uploadedFile)
}
I have a problem. I am using Team Foundation Server 2017 RTM. I have a build definition that will deploy my app to a development server running Windows Server 2012 R2. My app allows users to upload images and PDFs. When this is done, a folder named Media is created in my project's root directory and the files are uploaded here. The problem is, whenever I queue a new build, this folder gets destroyed and all the links to the media don't point to anything. I am rather new at managing and setting up TFS so I was wondering if there is any way I can preserve the contents of my media folder whenever I queue a new build. Any ideas?
Ok, so I spent my whole day looking at this.
In my C# code I create a directory like so:
// -- Create a new file name that is unique
string fileExtension = Path.GetExtension(upload.FileName);
Guid fileGuid = Guid.NewGuid();
string fileName = fileGuid + fileExtension;
// -- Create the directory and upload the image to that directory
string mediaDirectory = Server.MapPath("~/Media/");
Directory.CreateDirectory(mediaDirectory);
string filePath = Path.Combine(mediaDirectory, fileName);
upload.SaveAs(filePath);
I would then set the image url on the Media object like:
string imageUrl = "/Media/" + fileName;
So now, instead of storing the image in the database, I am just storing the URL to the image.
This was creating the directory in the app directory where I can store the files:
Which is cool but as I mentioned, this directory will be destroyed every time I queue a new build. How I fixed this was to modify where I stored the images:
// -- Create a new file name that is unique
string fileExtension = Path.GetExtension(upload.FileName);
Guid fileGuid = Guid.NewGuid();
string fileName = fileGuid + fileExtension;
// -- Create the directory and upload the image to that directory
// The Media directory will be created on the C drive root
string mediaDirectory = #"c:\Media";
Directory.CreateDirectory(mediaDirectory);
string filePath = Path.Combine(mediaDirectory, fileName);
upload.SaveAs(filePath);
Now my Media folder is created on the server's C drive and won't be destroyed whenever I queue a new build. Since the app can't access files outside the app directory, I needed a way to access those files in the Media directory. What I did was create a new virtual folder in IIS that points to the Media folder and gave it the alias Media:
This will now let me have access to all those files I put in the Media directory and will properly display the images when needed. I really hope this helps someone because I spent way too long looking at this.
According to your description, there is a concept of working directory in the build agent. If you set clean=true in the build definition, this will delete the previous build output when you query a new build. Not sure where you Media folder located, avoid to create/put it in some directory on the build agent such as Build.ArtifactStagingDirectory
The local path on the agent where any artifacts are copied to before
being pushed to their destination. For example: c:\agent_work\1\a.
A typical way to use this folder is to publish your build artifacts
with the Copy files and Publish build artifacts steps.
Note: This directory is purged before each new build, so you don't have to clean it up yourself.
More details about the folder path in build/release, you could refer this tutorial-- Predefined variables
In rails, and with (say 5k files) using the aws-sdk gem, what is the easiest way to copy a list of public files that are hosted on S3 (not my account) into my private bucket? I would want to keep the same file and path name.
Example:
http://target.com.s3.amazonaws.com/assets/videos/abc123.mp4 (public)
http://myexample.com.s3.amazonaws.com/assets/videos/abc123.mp4 (private)
I would like read the files into memory and directly stream into S3. I won't have disk space with my hosting provider (Heroku). These files are MP4s and are about 3-4MB in size.
Here's my approach (UNTESTED):
vid_file = 'http://example.com.s3.amazonaws.com/assets/videos/abc123.mp4'
vid_response = HTTParty.get(vid_file)
if vid_response.code == 200
filename = File.basename(vid_file) # TOOD - fix to include s3 folder before object filename
s3 = Aws::S3::Resource.new(region: ENV['AWS_REGION'])
obj = s3.bucket(ENV['S3_BUCKET']).object(filename)
obj.put(body: vid_response.body)
end
However, is the a way with the SDK to direct AWS to perform an internal copy between the S3 bucket, albeit I don't have the keys for the first bucket (but the objects are public)? If NOT, is my above approach correct (streaming into memory, posting to S3)?
One easy solution if you know the file name pattern is to use something like wget and then a ruby s3 client to upload to your private bucket. I understand why you would want to use memory instead of hdd but honestly assuming you have a couple gigs free your internet connection is probably the bottleneck.
1) There's is no sdk feature for an 'internal copy' of public S3 objects to ones private S3 bucket.
2) the below source works, which keeps the same S3 directory structure
vid_file = 'http://example.com.s3.amazonaws.com/assets/videos/abc123.mp4'
vid_response = HTTParty.get(vid_file)
if vid_response.code == 200
uri_path = URI(vid_url).path
uri_path.slice!(0) # slice!(0) removes leading slash, otherwise creates an empty s3 folder
s3 = Aws::S3::Resource.new(region: ENV['AWS_REGION'])
obj = s3.bucket(ENV['S3_BUCKET']).object(uri_path)
obj.put(body: vid_response.body) if !obj.exists?
end
I am trying to set up Weceem using the source from GitHub. It requires a physical path definition for the uploads directory, and for a directory for appears to be used for writing searchable indexes. The default setting for uploads is:
weceem.upload.dir = 'file:/var/www/weceem.org/uploads/'
I would like to define those using relative paths like WEB-INF/resources/uploads. I tried a methodology I have used previously for accessing directories with relative path like this:
File uploadDirectory = ApplicationHolder.application.parentContext.getResource("WEB-INF/resources/uploads").file
def absoluteUploadDirectory = uploadDirectory.absolutePath
weceem.upload.dir = 'file:'+absoluteUploadDirectory
However, 'parentContext' under ApplicationHolder.application is NULL. Can anyone offer a solution to this that would allow me to use relative paths?
look at your Config.groovy you should have (maybe it is commented)
// locations to search for config files that get merged into the main config
// config files can either be Java properties files or ConfigSlurper scripts
// "classpath:${appName}-config.properties", "classpath:${appName}-config.groovy",
grails.config.locations = [
"file:${userHome}/.grails/${appName}-config.properties",
"file:${userHome}/.grails/${appName}-config.groovy"
]
Create Conig file in deployment server
"${userHome}/.grails/${appName}-config.properties"
And define your prop (even not relative path) in that config file.
To add to Aram Arabyan's response, which is correct, but lacks an explanation:
Grails apps don't have a "local" directory, like a PHP app would have. They should be (for production) deployed in a servlet container. The location of that content is should not be considered writable, as it can get wiped out on the next deployment.
In short: think of your deployed application as a compiled binary.
Instead, choose a specific location somewhere on your server for the uploads to live, preferably outside the web server's path, so they can't be accessed directly. That's why Weceem defaults to a custom folder under /var/www/weceem.org/.
If you configure a path using the externalized configuration technique, you can then have a path specific to the server, and include a different path on your development machine.
In both cases, however, you should use absolute paths, or at least paths relative to known directories.
i.e.
String base = System.properties['base.dir']
println "config: ${base}/web-app/config/HookConfig.grooy"
String str = new File("${base}/web-app/config/HookConfig.groovy").text
return new ConfigSlurper().parse(str)
or
def grailsApplication
private getConfig() {
String str = grailsApplication.parentContext.getResource("config/HookConfig.groovy").file.text
return new ConfigSlurper().parse(str)
}