Related
I'm using Windows, Mac and Linux machines in my daily duties. On all machines, I program in C++ and various shells scripts. So far I've adopted the various "main" IDEs on each platform, but the diversity is irritating. I'm therefore looking into the possibility of using Sublilme Text on all platforms.
I have a setup of Sublime Text on Windows that works perfectly and would like to use the same on the other platforms also, so that when I change something in my Sublime setup on, say, my Mac, I can easily pick up the latest setup on my Windows machine the next time I'm there.
Is this possible on the 3 mentioned platforms, without getting (more) grey hair? If so, any suggestions or experiences thereof?
Many folks upload the "Packages/User" folder to GitHub (or your VCS service of choice). Then, they use Package Control to install their packages. Package control, through a settings file, will install any missing packages on a particular machine. I wrote a bit more about it here. You would then clone the git repo onto each machine, pulling updates when you decide to change something.
Alternatively, you could probably use a cloud service + symlinks to keep things auto synced, but I've personally never used it that way.
There are some plugins that are platform specific, so keep an eye out for those.
There's also the Package Package Syncing, which syncs installed packages and settings via some cloud service.
Works quite nicely, and automatically.
Has the advantage that you don't have to push/pull some dotfiles repo all the time.
No idea though whether this will work seamlessly across platforms (meaning whether all the settings will be platform-independent).
One exe file is system file, such as net.exe or sc.exe; one is downloaded like PsService.exe.
I found that One(net.exe or sc.exe) I can use it in a service via ShellExecute(); another cannot be used. what is different between them? any one knows?
simon
net.exe and sc.exe are system administration commands, and fire UAC if the current user session doesn't have administrative privileges. If you are trying to make changes to the system's configuration, please learn about WMI or the specific API for the sub-system (e.g. the Windows Network Functions) (as well as about administrative priviliges and how to obtain them).
We have a software package which is about 16 years old. It's gone through just about every version of Delphi (besides .NET ones). Over the years, things have become very confusing when it comes to cross-referencing and keeping proper setup for additional packages (like third-party libraries). I was wondering if there is some standard practice when it comes to keeping large projects (and groups of projects) like this organized.
So to explain the current setup...
This is a multi-application system. Meaning, there are 12 executable projects (and a few DLL and service projects) involved. We also keep things in SourceSafe and multiple developers work on the same code on different computers. All of these projects are more-so dumped into a central folder. The "Root" folder contains THE major EXE project (along with about 20 folders, all containing units and forms) and it almost seems like an endless hierarchy of folders and files. This one project alone has half a million lines of code involved.
Then all the additional applications aren't necessarily separated properly from this major project. Each of these projects has its own folder based in the main project's root.
The two major concerns of mine are:
How to properly set up the DCU files so that they aren't mixed in with the projects? DCU's should NOT be placed in the SourceSafe (and any similar file, for that matter) or otherwise, any file compiled from the project. Visual SourceSafe makes files read-only when they're not checked out, and DCU files (and EXE files and more) cannot be written to in this case. So how to properly separate any of such file to a remote location to avoid any mixture with the source code?
How to properly set up packages and libraries? We have the following:
QuickReports 5.05
NativeJpg library V302 -
Another anonymous reporting library
Our own component package, which requires QuickReports, NativeJpg, and the other anonymous library
All 4 of those libraries are stored in completely different places of each computer, and need some centralization. The biggest pain of setting up each new developer's computer is locating these from the lead developer's computer and copying them to the same place on each other computer (and making sure the library path is correct, etc.).
We also need to keep completely separate environments for different versions of Delphi on the same computer. This means a copy of the projects on each computer, a copy of packages and libraries on each computer, a copy of the projects and packages and libraries in the SourceSafe, etc. Each computer needs to have an identical setup. We already utilize environment variables to direct our projects where to look for certain project files (and libraries).
Another new concern: XE2 introduces 64bit capabilities. We don't plan on 64bit compiling yet, but we certainly will in the future. How do I properly differentiate 32bit from 64bit in all these projects?
What I'm really asking for is a reference to a good tutorial on how to optimize such an environment and keep it organized the best. I don't expect anyone to take the time and answer all this in the question. The projects are over 15 years old, have had the hands of 200+ developers from around the world in it, and has a LOT of cross-referencing between projects. For example, one project may use a unit from another project, and vice-versa. I personally don't like this concept, but I also didn't design it to begin with. I've been given the task to get this system organized and thoroughly documented how to set up Delphi on a new computer for new developers to work on our projects. As I'm looking at our projects (as I'm not necessarily a developer of the system, but am being pulled into development), I'm seeing a lot of confusion in how the code is organized.
I am assuming that possibly Embarcadero has some guidelines and standards on setting up such an environment?
Location of DCU files
Regarding the DCUs that are the output of the compilation process, you should specify a DCU output directory in each project file. The default value for this, in the latest version of Delphi would be fine: .\$(Platform)\$(Config). This results in sub-folders of the project directory like this: Win32\DEBUG or Win64\RELEASE.
If you set-up your project files using option sets then you will be able to control this setting (and all others) from a small number of option files.
Location of 3rd party code
You should always use 3rd party library as code. If the vendor charges more to receive the library as code, pay up. Once you have done so you simply include the source code into your version control system (VCS) and treat it largely the same way as you treat your own code. I say largely because you should avoid modifying it.
Once you have all your code in the VCS then you can put the entire source code onto a new machine with a single checkout operation.
Organisation of your projects
I personally have a strong aversion to using compiler search paths. I don't use them and include every unit that is required in a project in the .dpr file.
If you do use search paths then you make it impossible to work on variant projects.So for example, suppose you have a client that has discovered a bug in the version of the software you released 2 years ago. You would like to address that bug by releasing an upgrade to the 2 year old version of the software. It is perfectly plausible that asking them to upgrade to the latest version is not viable. Perhaps they have not paid for the upgrades. Perhaps the full upgrade has breaking changes that they do not want to tackle right now. A perfect example would be all the Delphi developers still using Delphi 7.
Now, having motivated the scenario, how would you create a build environment for the 2 year old project? If you are using search paths then they will refer to today's libraries. You would be forced to change your search path, or copy the old libraries over the top of today's libraries.
That entire headache is trivially side-stepped by not using search paths and by including all your source in the VCS.
What you should be aiming for is to be able to checkout any historic version of your program and have it build immediately. You should be able to do this with full confidence that you are building identical software to what was built at the time that version was released. This also requires you to have build automation but I can't imagine you are lacking that for a project of this size.
I'll address folder organisation. This comes from a software suite which has 50+ exe's and dll's and plenty third party libraries, so I guess I know where you are coming from...
We use Perforce as a source control system, so my default workspace's root folder is called Perforce, but I also have a couple of other workspaces set up and they are in Perforce2, Perforce3, etc.
General folder setup (starting from the workspace root folder)
General
Components
Delphi
Indy
Indy9
Indy10
MadCollection
v2.5.8.0
v2.6.0.0
Plugins
Releases
Released
... a folder for each release we publish ... (and equal to a branch in Perforce)
Work
Acceptance
Sub1
Sub2
My Environment library path in the IDE is empty (not even the BDE standard paths are in there). This ensures that a project's paths declare all path's needed and that projects are not reliant upon a particular machine's IDE setup.
We have an environment var (ie MRJ) set up in our IDE's that points to "General\Components\Delphi" so in a project's options we declare the paths to our components as $(MRJ)\MadCollection\2.6.0.0.
General holds IDE plugins and components used by our projects. We keep all versions we use in source control. That way when I have to switch back to an old release to track down a problem, I can simply pull it and build it as its library paths will still point to the version of the components that this specific release needs.
The organisation of folders in a particular work branch (Acceptance or one of its subbranches) follows this pattern:
General
Includes
MainComponent1
Project1
Project2
Shared
MainComponent2
Project3
Project4
Shared
Shared
Windows
SoftwareSuite
Scripts
Tools
MainComponent1
Project1
Dcus
Project2
Dcus
MainComponent2
Tools
Tool1
Dcus
Tool2
Dcus
The General folder holds all platform independent sources/files, the Windows folder holds all Windows specific files. Each component can hold multiple projects and will have a share folder for sources shared between those projects. The shared folder directly under General holds sources shared by all projects. The Windows folder is set up in a similar manner.
Note that each project has its own dcus folder. This is configured in the project options. As the path can be entered as .dcus, we (at least I) have this set up as the default for any new project. Each project sending its dcus to a unique folder ensures two things:
it is easy to keep dcu's out of version control by simply setting up a filter in your version control software.
more importantly it ensures that compilation/build of a project never interferes with the compilation/build of another project. I can safely change settings and build knowing that I won't be bothered by dcu's lying around from a previous build from another project.
I recommend the following practices:
Keep your library path simple, and make sure everything in the library path is either a folder that ships with delphi, or a DCU binary (library) folder in your d:\Components\ folder.
Use a MODERN type of version control. I recommend Mercurial over others. Source Safe is crap, stop using it.
Back up your environment (export registry keys etc) and restore it to the other developer PCs in a standardized way. You can keep a few .reg and .cmd (batch) files around to automate setup of a new system. you can put these scripts in your component repository in your version control system.
Outside the scope that was largely discuss before, I would recommend :
Unit testing - with DUnit for example
Continuous integration. Just to be sure that all these projects can compile on another machine and that tests are ok.
So this is heavily related to project organization and VCS strategy.
For a similar setup, a company I worked for found this configuration useful:
all third party libraries (components etc.) go to a fixed location (C:\Delphi\name-version)
Delphi projects can be checked out from version control anywhere (drive C: or D: and folder name does not matter), as all projects and scripts use relative paths
all projects are sub folders of one main project folder so checking out this one will bring the Delphi projects and other relevant resources to the workstation, and a version control update is easy to do
we use a build script (written in Apache Ant) which sits in the main folder, and iterates over all folders to build the Delphi apps and run unit and integration tests against a development database server, to verify all changes work before checking in to source control
the build script can also be run automatically on a build server (Hudson CI) on every commit to see if something broke
And a note about component libraries: avoid package installation where possible, prefer creation of components at run time. If you quickly need to apply a fix to a five year old version of a project, uninstalling / installing a dozen of packages can become frustrating. At least for non-visual components, run-time creation is a huge time saver.
Checking in third party code in source control can be very helpful, for example to share fixes which are not yet available as new official releases. Best practices are covered in the Subversion documentation chapter Vendor Branches.
Plus, with Subversion you can use svn:externals to place a specific version (tag) right into a project directory structure. This can be used both with third party library and with your own source code, and makes dependency management easier and workstation setup easier.
p.s. the Ant build script defines the search paths for everything, so it is 'the reference' for all developers how to configure the IDE, where to put the third party libs and which compiler flags to use
p.p.s. your project sounds like a lot of fun - I am open for contract work :)
My team use virtualization and when we see back it was a real good move.
We use MacBook Pro laptops and VmWare Fusion, but I'm sure other packages work fine as well like VirtualBox or VirtualPC.
It is always a good feeling to know that when a new developer starts or an old installation got trouble it is just to copy a new VM image from the master image and the setup is exactly as the original. The master image is stored on a fast USB2-disk. Now when Thunderbolt and USB3 is coming it would be even faster to copy an image. And there is no real concern about performance on a modern computer as long as there is memory. 8 GB should be enough to run 2 images in parallell. Another advantage of virtualization is that it is so easy to try What if scenario. Experiment with different configuarations and versions without any risk to disturb the real working environment.
Btw I also think that SourceSafe is crap... :-)
Somé tips:
Make one groupproject file for all the apps belonging to the project, each app in its own dir under the groupproj file
You should be able to specify which file types to include into your version control system. Make sure you set Delphi to write DFM files in text format.
You could tell Delphi to output DCUs in subdirs named 'dcu' under each app (less visaul clutter).
Third party stuff often insists on installing in distinct locations, there's not much you can do about it. Make a document describing how to setup a complete working environment and keep it up-to-date
Develop in virtual machines. A new developer gets a copy of the VM.
Maintaining for different Delphi versions? Rethink that, try to go to one version. If you absolutely must have two groupprojects and directory structures for each version. [I'm assuming you're not compiling the same app with two Delphi version, that's developer hell]
Delphi XE2 will output to different 32/64 subdirectories, that should give no problems.
We are getting familiar with Erlang/OTP and Riak (Core, KV, Pipe, and so on) trying to build an open-source distributed application in turn. Our project is going to have a bunch of dependencies : tools like erlang_js, protobuffs, etc., but also services that we need to have running, such as Riak KV.
In Python/Ruby/Node.js, if modules are placed in a standard subdirectory relative to your project's, you can reference them, and later package them in releases. You can fire up a shell in the project's directory, play with your modules, do tests and so on, all just easily if good practices are followed.
What are the best practices for organizing a development environment in Erlang/OTP, with all dependencies reachable (and easily updatable to newest version), shell access to running nodes, testing, making releases, and so forth?
Check out rebar3: used for packaging, templating and managing releases of Erlang/OTP applications. You will find there an entire tutorial on how to use it.
Check out this tutorial on OTP first, before you can switch to this one which shows you releases and how they are handled in Erlang. Use this entire book as reference as you develop your project and because the author is still progressively adding more advanced stuff that you may need.
You may also want to keep reading about Erlang Applications and then do check out these quick links below:
Erlang Packaging, Process OneOTP Packaging Video by Chicago Boss GuysErlang Dependency ManagementRichard Jones Advice and examples on Packaging Erlang Projects with DependenciesMaven Tool and how it is used in packaging Erlang Projects
Most importantly, take rebar very seriously. It is very important in managing Erlang applications which have dependencies. You can keep posting your questions here on Stack Overflow for any assistance as you progress.
Before I forget, do check out a lot of stuff from the Riak Community.
Also, its important to check out the system documentation on creating target systems and also how to use Reltool to handle releases. The good thing with erlang is that it has several options of how to do something, as long as its easy to maintain your application in that way. With target systems, you will learn how to embed the Erlang VM, how to run Erlang applications on Solaris, VxWorks, and creating Erlang applications as Services on Windows NT using erlsrv.
Normally, we make sure that as an operating system is booting a Server, our application starts with it. Solaris so far has more customizations than any other OS as regards embedding Erlang/OTP VM. You can always communicate with (an) embedded Erlang VM(s) using Escript where by the escript creates a an erlang node which is allowed to connect to the embedded VM (so they have to share same cookie) and the embedded VM must have allowed the temporarily created Node to connect by executing net_kernel:allow(List_of_nodes)
Make sure to call this method to make your embedded VM allow connections only from an known strictly specified number of Nodes.
Apache Maven is a very popular build and dependency management tool in the Java open source ecosphere. I did some tests to find out if it can handle compiled Free Pascal / Delphi units and found it easy to implement. So it would be possible to
release open source libraries precompiled for Free Pascal (or Delphi) in a public Maven repository
include metadata in this repository which contains dependency information
use Maven on the command line to download the open source library from the public repository, and automatically resolve all dependencies
local repositories, working as proxies, could be used to cache frequently used binaries
automatic checksum generation and verification (provided by Maven) would reduce the risk of downloading corrupted binaries
source code and even documentation files could be provided with the binaries
binaries can be provided with or without debug information
continuous integration servers like Hudson, TeamCity or CruiseControl can be used to build projects whenever changes have been submitted to the source control system and notify developers about build errors
This way of dependency management could be very beneficial for open source projects which use many third party libraries with complex dependencies. It would avoid typical conflicts caused by using wrong versions.
For the developer, the workflow for editing and building a project would be reduced to a minimum:
checkout the project source from internal version control system
edit source file(s)
run mvn package to automatically download all required third party libraries (precompiled units) if they are not yet in the workstation's local repository
compile and run
The only additional file for Apache Maven which is required in the project folder is the POM.XML file containing the project information.
Edit: while Maven is usable for some of the required tasks, implementing a solution like Maven in native Free Pascal would have some advantages: no Java SDK required, support for all development platforms where Free Pascal is available, maintenance and plugin development in Pascal.
Usage of a Maven-like tool would not be helpful for open source projects only - commercial projects could access and use the artifacts in public Maven repositories in the same way as well.
Maven features are listed at http://maven.apache.org/maven-features.html
Update:
one use case could be the build of Lazarus, where Maven would download all required libraries and invoke the compiler with the necessary build path arguments. Changes in the dependencies on lower levels would be propagated automatically up to the parent build.
Possible benefits:
less time needed to set up a new work
station, no manual installation of
third party libraries required
less errors caused by wrong library
versions, detection of version
conflicts (for example if two
libraries depend on different
versions of a third library)
artifacts which are created inhouse
can be added to the local maven
repository and shared between
developers and project, central
storage of all artifacts with
metadata
builds are reproducible, just by
using the same source and project
metadata file (pom.xml)
can reduce development time and
increase project stability
Update #2: FPMake
the FPMake build system for Free Pascal seems to be a tool with much potential, in many details it is quite similar to Maven:
FPMake is a pascal based build system developed for and distributed with FPC
FPMake standardizes the building by defining some limits like standard directories
the command fppkg <packagename> will look in a database for the package, extract it, and then compile fpmake.pp and run it
it has standard build targets (clean, build, install, ...)
it can create a 'manifest' file suitable for import into a repository (like mvn deploy or mvn install), the manifest is an XML file which looks very similar to a pom.xml in Maven:
FPMake manifest file:
<packages>
<package name="my-package">
<version major="0" minor="7" micro="6" build="1"/>
<filename>my-package-0.7.6-1.zip</filename>
<author>my name</author>
<license>GPL</license>
<homepageurl>http://www.freepascal.org/</homepageurl>
<email>myname#freepascal.org</email>
<description>this is the package description</description>
<dependencies>
<dependency>
<package packagename="rtl"/>
</dependency>
</dependencies>
</package>
</packages>
Freepascal has been working on a package system of its own in a cross between apt-get and freebsd ports style. (download source/build/install automatically), called fppkg.
However work has stalled. People investing time are the bottleneck, not people wanting to choose tools.
As far as Maven goes, I don't like auxilary tools that need installation of huge external runtimes. It might be fine for a big major app (like Open Office), but not for an util.
I also prefer a tool that is designed to the FPC reality and workflow.
Documentation tools, build tools, download systems, testsuite systems are already all there, it just need a person that dedicates a lot of time into it to make it happen.
Some typical problems when introducing a new technology in a project as FPC, and why it has a tendency to make its own tools:
need to train 20+ committers in parttime.
The only COMMON programming language you can assume is Free Pascal. Even Delphi inner workings can't be taken for granted to be known (many committers came directly to FPC or even still via TP or a Mac Pascal)
Obviously that makes something with plugins in a different language annoying.
Bash script is a close second. (g)make third, but already a magnitude less.
All servers are *nix-like (FreeBSD, OS X, Linux), but not all run Apache. (e.g. my FreeBSD mirror runs XSHTTPD)
somebody most knowledgable must be dedicated maintainer for a long time. Fix problems, update/ do migrations etc. Perferably more than one for obvious reasons.
a major pain are Linux distributions (and FreeBSD to a lesser degree), most maintainers of *nix packages are not capable of more than "./configure;make;make install", and must be spoonfed with a near buildable repository and auxilary files.
In-distribution packaging of FPC/Lazarus has always been important, and is still increasing
All distributions have their own special rules about metadata, depedancies, and how sources must be published. Particularly Debian/Ubuntu is very bureaucratic and slow.
Most don't like third party auto-installers on top of their systems (since that bypasses their dependancy control)
This all leads to the effective practice that own tools in Pascal with minimal scripting work best. Some tools used:
Gmake is mainly used to parameterise the build process on a per directory level, a successor, fpcmake (not really a make derivative despite the name) has begun, but the migration hasn't completed.
Latex and a latex to html conversion (tex4ht, but debian uses hevea) are used in the documentation building (the non library documentation)
The community site (netscape community server which uses TCL scripting, a heavy complex application server) has been a trouble ever since it started, but specially lately since the maintainer became less active.
Mantis has been a problem (specially the email module would crash or lame the server due to the volume), but it has been whipped into shape during successive updates and hard work of several lazarus devels. Currently it is a decent workhorse.
lazarus.freepascal.org PHPBB forum OTOH is relatively painless since a lot of younger people know how to deal with it.
The same goes for subversions (though the more advanced scale needs some adjusting, not everybody is deep into the ins and outs of mergetracking)
If somebody was really serious about Maven, I usually would ask him:
to CRITICIALLY investigate the use for the project. In a very concrete way, with schedule and time estimates. Birds-eye level "everything's possible" overviews are essentialy worthless.
Give some thought on future change of used technologies. Every technology is eventually replaced, even the in-house ones, in 18 year+ projects. A new technology must not make migrations of other infrastructural components hard or involved. The new technology to end all new technologies doesn't exist.
Make a migration plan. Migration is often underrated and underestimated.
And in the end, there is always the 1000000 Euro question, who will do the daily maintenance?
Keep in mind that in a company you just kick the person responsible for the application server. But in an informal environment this is way harder, specially long term, since people's lives, occupations and time spent on the project vary.
Sounds like an interesting plan, but the Delphi community (and FPC even more so, I'd imagine!) values libraries as source far more than precompiled libraries. The general consensus is that anyone who uses a binary-only library is a fool, for two reasons: You can't fix any bugs you find in it, and compiler changes will break compatibility.