Wishlist

Revision as of 14:02, 23 November 2006

Separate production data from the WebApp directory

OpenCms writes to its web application context directory (for Tomcat, that is $CATALINA_HOME/webapps/opencms). To be precise, the problematic fact is not that OpenCms writes to a directory but that it mixes deployed files and production data.

OpenCms comes in a WAR file which is impossible to be used like it is meant for WAR files. You cannot simply deploy it, you have to unpack it because OpenCms checks that it can write to the WebApp directory. During setup and later OpenCms writes production data to this directory. This makes it impossible to use standard J2EE procedures (e.g. redeploying the WAR file when you changed something) and it makes development of solutions which integrate and extend OpenCms very uncomfortable.

Suggestion

This wish proposes that OpenCms should be deployable and redeployable without loss of configuration or production data from an unpacked WAR file, even if the Servlet Container is configured such that the WAR file will not be unpacked. Configuration data and production files should be outside of the WAR file, in a directory called OPENCMS_HOME. Files exported from the OpenCms VFS which need to be HTTP-accessible via the Servlet Container should be in a separate Webapp to which OpenCms is allowed to write.

This separation of code and data would make the development and deployment process compliant with what developers are used from J2EE:

• Take opencms.war as the basis to start a custom development
• Develop code, JSP, etc. in an IDE
• Package it in a WAR file
• Deploy the WAR file to the container
• Run OpenCms Setup (only once!) to create and populate OPENCMS_HOME.
• Modify the WAR file
• Redeploy it to the container

With the current OpenCms, one has to do the following:

• Extract opencms.war to the deployment directory of the Servlet Engine
• Run OpenCms Setup
• Develop code, JSP, etc. in an IDE
• Copy the files by hand (or with the help of a script) to the directory where OpenCms is deployed

WAR files are the unit of deployment for web applications and they should continue to be even for customized versions of OpenCms.

The current solution is focused on out-of-the-box installations where development only uses the tools provided by the OpenCms frontend and takes place only after setup has been performed. However, almost no development effort is usually done this way. Redeployment of the WAR file without losing configuration or production data is an important requirement which OpenCms does not fulfill at the moment. This is due to the fact that OpenCms writes to its webapp directory.

Analysis

I will first try to list all situations where OpenCms does this and explain which problems are caused by those write accesses. Afterwards I will propose several ideas on the solution to these problems and finally detail on the gains.

As of version 6.2.2 OpenCms writes to files on disk in the following situations.

1. After installation OpenCms disables the setup code in WEB-INF/config/opencms.properties.
2. OpenCms changes configuration files in WEB-INF/config/ and takes backup copies of these in WEB-INF/config/backup.
3. OpenCms writes its log files to WEB-INF/logs (setup.log, opencms.log)
4. OpenCms writes often needed workspace files to resources/ and static export files to export/
5. It writes exported module ZIP archives to WEB-INF/packages/modules
6. It writes module ZIP archives uploaded via HTTP to WEB-INF/packages/modules
7. When publishing a module, OpenCms writes the module's classes and classpath resource files to WEB-INF/classes and the module's libraries to WEB-INF/lib.
8. It writes image data to WEB-INF/imagecache
9. On updates to search index files in WEB-INF/index
10. When JSP files are published from the VFS, they end up in WEB-INF/jsp

Problems

Problems caused by OpenCms mixing code and production data:

1. First and above all, updating the web application is complicated and error-prone. It is impossible to just redeploy the opencms.war file. This would destroy the installation.
2. You need to reload the OpenCms context or even restart Tomcat when you make classpath relevant changes to a module. This makes testing and bug fixing very inefficient.
3. It is impossible to include opencms.war as part of a J2EE EAR file.
4. It is impossible to package and deploy a WAR file which contains the original or modified contents of opencms.war.
5. It is very difficult to set up a development environment and a build process for OpenCms module development which includes an IDE and a version control system.
6. It is quite unprobable but possible to get errors almost impossible to track down if two modules have conflicting versions of files, classes, or libraries (same path or conflicting versions). Publishing of the modules is a last-one-wins situation.

Details of the proposed Solution

The following measures would improve the situation:

1. Separate the log files, indexes, cache and configuration data from the OpenCms context. This could be a standard location, for example /home/opencms or a similar directory with sub folders like config/ and logs/. This measure is concered with write situations 1, 2, 3, 8 and 9. Let us call it OPENCMS_HOME.
2. Separate resources/ and export/ from the OpenCms context and put them into a second web application context to which OpenCms is allowed to write. This measure is concered with write situation 4.
3. Since the module ZIP archives in WEB-INF/packages/modules/ are not used for any purpose after setup they should be removed completely from RFS. OpenCms should load the module classes, classpath resources and libraries with a custom ClassLoader for each module directly from the VFS. This measure avoids write situations 5 to 7. (The module ClassLoaders would build on the defined module dependencies.)
4. OpenCms should load JSPs directly from the VFS using a custom ClassLoader. (write situation 10)
5. There could be a 'hot module folder' watched by OpenCms for updates just like the deploy/ folder with JBoss, e.g. in $OPENCMS_HOME/modules. (optional)

Benefits

• Simple redeployment/upgrade of opencms.war without fear of breaking the installation.
• You could simply include opencms.war in a J2EE EAR file.
• One could easily customize opencms.war and deploy the customized version
• Backup/replication of OPENCMS_HOME as well as the writable webapp context is still straightforward.
• Simple development/testing of modules without reloading of the OpenCms web application. OpenCms would just drop the old module ClassLoader and reload the new module version.
• Easier start for developers new to OpenCms.

Custom ClassLoaders for Modules

OpenCms should provide its own ClassLoader for each module.

• The Module ClassLoader is instantiated along with the module and used to load classes, classpath resources and libraries contained in the module directly from VFS.
• The classpath of a Module ClassLoader contains
  • /system/modules/<modulename>/classes/
  • /system/modules/<modulename>/lib/*.jar
• Module dependencies make up a Module ClassLoader hierarchy. If module B depends on module A, the Module ClassLoader of B has the Module ClassLoader of A as its parent, thereby making A's classpath available to B.
• The Module ClassLoader of a module which does not depend on a different module has the standard Webapp ClassLoader as its parent.
• When a class should be loaded, the Module ClassLoader first looks into the module classpath. If it does not find the class there it delegates class loading to its parent.
• Each time OpenCms detects that a file on the Module ClassLoader classpath changes, the Module ClassLoader is thrown away and a new instance is created.

Benefits:

• No need to export files from the VFS to WEB-INF/lib and WEB-INF/classes.
• It is no longer required to reload OpenCms for each change to a module classpath file.
• Ability to use different and incompatible versions of libraries in different modules.

Open Questions: How would the Module ClassLoader perform JSP loading?

manifest.xml

The file manifest.xml contains information about a module. However, as of version 6.2.2, there is no way to create such a file outside of OpenCms which is an obstacle for the creation of modules for OpenCms.

If a ZIP archive which contains module files does not have a manifest.xml OpenCms refuses to import the module. This makes module development hard. You need to use the synchonization feature which is not appropriate in all cases because it cannot distinguish system files and files added by a module (e.g. the icons in /system/workplace/resources/filetypes) and thus can lead to inadvertent deletion of system files if used without enough experience.

There is currently only one way to create a valid manifest for a module: Make all changes in the OpenCms frontend and export the module. There should be a way to create a valid module ZIP archive outside and without the help of OpenCms.

Wishes:

• The XML Schema Definition of the manifest file format of should be made public
• There should be a small tool independent of OpenCms with an API to create and maintain manifest files for modules being developed outside of OpenCms.

JSR 170

It would be good for OpenCms if it would implement the JSR 170. In the spirit of openness and interoperability, this would be a major boon.

More information on JSR170

Article on CmsWatch providing a nice overview of the JSR

The JSR on the JCP site