Deploying native libraries with TFS deploy agent - tfs

Question

I'm using libgit2sharp in my project which uses Git2 library to work. Now, the last version used project properties files instead of custom build action. While this work fine when building or using the Publish action in Visual Studio 2013, it won't work at all using TFS build server.
Here are the imports instruction in the csproj file.
<Import Project="..\packages\LibGit2Sharp.0.20.0.0\build\net40\LibGit2Sharp.props" Condition="Exists('..\packages\LibGit2Sharp.0.20.0.0\build\net40\LibGit2Sharp.props')" />
<Target Name="EnsureNuGetPackageBuildImports" BeforeTargets="PrepareForBuild">
<PropertyGroup>
<ErrorText>This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}.</ErrorText>
</PropertyGroup>
<Error Condition="!Exists('..\packages\LibGit2Sharp.0.20.0.0\build\net40\LibGit2Sharp.props')" Text="$([System.String]::Format('$(ErrorText)', '..\packages\LibGit2Sharp.0.20.0.0\build\net40\LibGit2Sharp.props'))" />
</Target>
And here is the props file in question:
<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<None Include="$(MSBuildThisFileDirectory)\..\..\lib\net40\NativeBinaries\amd64\git2-3f8d005.dll">
<Link>NativeBinaries\amd64\git2-3f8d005.dll</Link>
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Include="$(MSBuildThisFileDirectory)\..\..\lib\net40\NativeBinaries\amd64\git2-3f8d005.pdb">
<Link>NativeBinaries\amd64\git2-3f8d005.pdb</Link>
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Include="$(MSBuildThisFileDirectory)\..\..\lib\net40\NativeBinaries\x86\git2-3f8d005.dll">
<Link>NativeBinaries\x86\git2-3f8d005.dll</Link>
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Include="$(MSBuildThisFileDirectory)\..\..\lib\net40\NativeBinaries\x86\git2-3f8d005.pdb">
<Link>NativeBinaries\x86\git2-3f8d005.pdb</Link>
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
</ItemGroup>
</Project>
So, again, publishing or building the project with Visual Studio will copy the NativeBinaries folder to the bin folder which is perfectly fine. But, using a TFS build deployment setup, the folder is never transfered over the server using the RemoteAgent.
MSBuild arguments:
/p:DeployTarget=MsDeployPublish /p:Configuration=MEP-DEV /p:DeployOnBuild=True /p:CreatePackageOnPublish=False /p:MsDeployPublishMethod=RemoteAgent /p:AllowUntrustedCertificate=True
In the build log file, I can see that the Libgit2sharp NuGet package is being installed, but the libraries are not mentioned anywhere as if they are just ignored by the deployment process.