=============== 1. Introduction =============== 1.1. Project/Component Working Name: Berkeley DB 4.7.25 1.2. Name of Document Author/Supplier: Karen Langford (karen.langford@sun.com) Lars Heill (lars.heill@sun.com) 1.3. Email Aliases: 1.3.1. Responsible Manager: yngve.svendsen@sun.com lars.heill@sun.com 1.3.2. Responsible Engineer: karen.langford@sun.com 1.3.3. Marketing Manager: 1.3.4. Interest List: karine.excoffier@sun.com matt.wall@sun.com ================== 2. Project Summary ================== 2.1. Project Description Currently, the Berkeley Database (BDB) ships within a number of Sun products. This ARC case is to establish BDB 4.7.25, the latest version of the Berkeley Database, as an accepted database technology for product use within Sun, including Solaris, and the Sun Java Enterprise System. The Berkeley Database is third party software owned by Oracle. We have a commercial license and support contract with Oracle to use the BDB source. We do no development, just RE work: compile, package and deliver the bits to our Sun internal customers, as well as integrate into Nevada. Our group has ARCed the BDB versions 4.2 and 4.5. Packages containing BDB 4.5 and 4.2 were integrated into Nevada in b91 (May 08) and made available internally to product teams. In the interim, Directory Server and Messaging Server approached us with a request for the 4.7 release. Both teams have strong business cases as to why they require 4.7 for their next product releases. This project proposes replacing BDB 4.5 with 4.7 and integrating into Nevada. We will continue to deliver BDB 4.2 to support those customers not willing to update. We are targeting Nevada b100 (10/06/08). BDB 4.7 is a minor update release. 2.2. Previous relevant ARC cases LSARC/2003/585 BDB 4.2 - Integrated into Solaris 10 and Nevada LSARC/2007/299 BDB 4.5 - Integrated along with BDB 4.2 into Nevada Note BDB is not in OpenSolaris. =================== 3. Business Summary =================== 3.1. Problem Area: We do not fore see problems with replacing BDB 4.5 with 4.7, as Evolution is the only BDB 4.5 user in Nevada todate, and they have been notified of the version change. 3.2. Requester: The requester is the Database Technology Group on behalf of the Messaging Server, Communications Suite and Directory Server product teams. 3.3. Business Justification: Copy of business justification sent to DBTG from Karine Excoffier (Engineering manager for Directory) and Matthew Wall (Engineering manager for Messaging - Communications Suite) Let me explain why Messaging Server, Communications Suite and Directory Server require the latest release of BDB to improve concurrency and performance. Messaging Server and Communications Suite do need concurrency and performance in the message store that is the heart of this product. BDB forms a critical part of the internal embedded message store and is fundamental to the performance requirements. Messaging customers are among the largest service providers (and biggest Sun software and hardware customers, including one that is billed over $1B annually) and have long been demanding optimal performance in message database transactions. The differences between performance in this space between earlier versions and the latest release are noticeable. Messaging team had hoped to have 4.6 included in its major revenue release scheduled for May 5th, and the fact that it is not in there will be a problem with these customers. Performance is also critical for Directory Server which consistently have to propose better performance to its customers (NSN, Sprint...). The next major release is planned in March 09. In addition, there are a number of other bug fix issues related to the particular version numbering, of course, including some fixes to regressions that other key customers have been waiting on for some time, but the business-critical issue is the enhanced performance we require to maintain our largest customers' business. We cannot afford to be trailing-edge in this area. 3.4. Opportunity Window/Exposure: We would like this ARC case approved by October 01, in order to make Nevada b100 and for Directory Server to incorporate BDB 4.7 into their minor release due out later this year and allow them to fully port their major release due in March 09. ======================== 4. Technical Description ======================== 4.1. Details Berkeley DB is a C-based, record-level database package. It allows very fast, vertically scalable, transactional access to persistent data. It is already in heavy use within Sun; in particular it is the underlying data store for Sun Java Directory Service. We will follow guidelines for package delivery and release versioning as agreed upon in LSARC 2007/299, including the continued delivery of the 4.2 release. Our customers, namely Directory Server and Messaging Server, are most interested in the bug fixes, and performance improvements introduced by the 4.7.25 release. Summary of new features are: + Numerous bug fixes + Major improvements to performance + Support for new platforms (WinCE, S60, Qualcomm's BREW) + Addition of verbose output tracing for filesystem operations. + Configuration enhancements to locking, logging and replication functions resulting in improved performance. + Addition of Java Direct Persistence Layer API which provides a mechanism for storing and managing Java class objects in a BDB database. For more information on the new features of BDB 4.7.25 please see http://www.oracle.com/technology/documentation/berkeley-db/db/ref/changelog/4.7.html http://www.oracle.com/technology/documentation/berkeley-db/db/ref/changelog/4.6.html 4.2. Bug/RFE Number(s): 6723517 4.3. Upgrading 4.3.1. Releases The majority of our customers still use the 4.2 release and have not migrated to 4.5. We will continue to deliver 4.2 for customers not ready to move to 4.7 in a transistion period before 4.2 is EOLed. Note the BDB software does not automatically inform applications that a new release is available. The decision to upgrade must be made by the customer. Existing applications will continue to access the old library (4.2) until the user performs the manual steps required to upgrade which include linking to the new library and recompiling. Installation of the new packages will not overwrite the existing 4.2 install, the new 4.7 version will install into a separate directory .../bdb47/... See the attached pkgmap for detailed layout. The issues regarding the delivery of multiple libraries, 4.2 and 4.7, and the upgrading procedures are the same as presented and discussed at the 4.5 ARC commitment hearing: http://sac.sfbay/arc/LSARC/2007/299/20080205.2007.299.commitment.mp3 Note our group plans to depreciate the 4.2 release in the near future. 4.3.2. Upgrading/Compatibility Before performing an upgrade customers must backup their database and log files. The upgrade process for the Berkeley database requires that customers perform a number of manual steps. These steps are clearly documented in Chapter 31 of BDB's Reference manual and must be followed by the customer. http://www.oracle.com/technology/documentation/berkeley-db/db/ref/toc.html The Berkeley database does provided a db_upgrade utility which must be used as part of the upgrade procedure. The 4.7 release introduces a number of modifications to existing APIs in the form of method name changes, and new parameters. These incompatibilities with the previous release may be transparent to a customer's application, depending on how the application is coded. An example of an API change in moving from 4.5 to 4.7: DB Cursor APIs were renamed from (for example) dbc->c_close() to dbc->close(). All code referencing these APIs would have to be updated. [2] As part of the upgrade procedure each customer must determine whether their application requires modification based on changes introduced by 4.7. Again, this information is available in BDB's reference manual. 4.4. I18N/L10N Impact: NA 4.5. Packages SUNWbdb - Berkeley DB for C/C++ applications SUNWbdbj - Berkeley DB for Java applications 4.5.1. Packaging & Delivery: For Solaris and Nevada: Cluster id: SUNWCbdb Metaclusters: SUNWCXall, SUNWCall, SUNWCprog, SUNWCuser, SUNWreq Cluster name: BerkeleyDB-Base Packages: SUNWbdb, SUNWbdbj We will update packages with the new release. These packages will not overwrite an old install of BDB. 4.5.2. Package Layout BDB 4.2 is currently installed under: /usr/lib/bdb - c utilities /usr/lib - libraries (versioned ie: libdb.so.1, libdb_java.so.1) /usr/share - db.jar We propose to install BDB 4.7 under: /usr/lib/bdb47 - c utilities /usr/lib - libraries (versioned ie: libdb.so.5, libdb_java.so.5) /usr/share/bdb47 - db.jar The case materials also includes the list of related pkgmap files for all installed packages. [1] 4.6. Security Impact: NA 4.7. Interfaces The Berkeley Database is embedded software. The Berkeley Database is a third party product and according to our licensing agreement the interfaces cannot be made available directly to the customer. This has not changed. 4.7.1. New Components Two new components have been added to the 4.7 release. - Java Direct Persistence Layer(DPL) API. This API provides a mechanism for storing and managing Java class objects in a BDB database. http://www.oracle.com/technology/documentation/berkeley-db/db/java/index.html - db_codegen utility. Generates application code to create and configure Berkeley DB database environments and databases based on a simple description language. http://www.oracle.com/technology/documentation/berkeley-db/db/utility/db_codegen.html 4.7.2 Removed Components None. 4.7.3 Interfaces added EXPORTED INTERFACES: Proposed Former Stability Interface Stability Classification Name Classification and Other Comments ------------------ ------------- -------------- -------------------- Java DPL API Volatile External db_codegen Uncommitted Unstable See Appendix A for full BDB 4.7.25 Interface table. ====================== 5. Reference Documents ====================== Berkeley DB Public Websites: http://www.oracle.com/technology/products/berkeley-db/index.html Berkeley DB 4.7.25 Release Notes: http://www.oracle.com/technology/documentation/berkeley-db/db/ref/changelog/4.7.html http://www.oracle.com/technology/documentation/berkeley-db/db/ref/changelog/4.6.html References: [1] ./pkgmaps.txt [2] ./low_level_api_changes.txt ========================= 6. Resources and Schedule ========================= 6.1. Projected Availability This project will be included in Solaris Nevada. This project will not be included in S10, note S10 does include BDB 4.2. This project will not be included in OpenSolaris. 6.2. Cost of Effort Estimated at 2 person months. 6.3. Cost of Capital Resources Negligible. 6.4. ARC review type: [Standard/FastTrack/SelfReview] FastTrack ========================= 7. Prototype Availability ========================= 7.1. Prototype Availability BDB 4.7.25 binaries are available to test. 7.2. Prototype Cost Minimal. No changes to packaging has occurred. Appendix A FULL LIST of EXPORTED INTERFACES in BDB 4.7.25 : Proposed Former Stability Interface Stability Classification Name Classification and Other Comments ------------------ ------------- ---------------------------------- C API Volatile External C++ API Volatile External Java API Volatile External Java DPL API Volatile External TCL API Volatile External Repl API Volatile External DB_HOME env Volatile External, Used to configure the variable location of db files. DB_CONFIG Volatile External, Used to configure a config file database database file Project Private transaction log Project Private temporary backing Project Private files db_archive Uncommitted Unstable db_checkpoint Uncommitted Unstable db_codegen Uncommitted Unstable db_deadlock Uncommitted Unstable db_dump Uncommitted Unstable db_hotbackup Uncommitted Unstable db_load Uncommitted Unstable db_printlog Uncommitted Unstable db_recover Uncommitted Unstable db_stat Uncommitted Unstable db_upgrade Uncommitted Unstable db_verify Uncommitted Unstable