PSARC Questions Version 1.17 Here is a comprehensive list of questions that ARC members may ask of project presenters. Providing this information in advance of your review will greatly simplify the ARC's process of identifying the critical information relevant to your project. It is expected that many of these issues may be unresolved at the time of an inception review. However, they should be answerable at commitment review, and may be addressed at inception if significant. Please make your answers concise! Most if not all of these questions will be addressed in related documents such as 1-pagers, specs, design documents, etc. There is no reason to duplicate effort, and "see section 3.2 in the design spec" is an excellent answer. Of course, the referenced material must be provided with your submission. Entire sets of questions may be N/A for your project. For example, device drivers rarely have GUIs, and so the entire GUI section can just be deleted. In such cases, PLEASE NOTE N/A FOR THE MAIN QUESTION, AND DELETE THE REST OF THAT QUESTION SET. This questionnaire is meant to provide the ARC with an overview of your project, and it touches on the main areas of architectural interest. This template will be revised based on its users' experiences; your comments and suggestions are welcome. Send them to john.plocher@sun.com. For advice about architectural choices, pointers to various SAC guidelines, and other project considerations including Licensing and Patents, see http://sac.sfbay/arc/ARC-Considerations.html ------------------------------------------------------------------------ 1. What specifically is the proposal that we are reviewing? The DMM project is producing an open-source tool which facilitates the movement of file systems from one platform to another over the NFS and/or CIFS protocol. Read/write access to file system content is maintained over the duration of the migration. - What is the technical content of the project? See DMM Functional Spec section 1.1 - Is this a new product, or a change to a pre-existing one? This is a new project formed from the remains of two previous projects. - If your project is an evolution of a previous project, what changed from one version to another? DMM has evolved from a data migration project code-named Hoover and a recently created data migrator being done in the services area also called DMM. The QFS-based kernel component of Hoover was eliminated and the DMM kernel component was refitted to use FEM and both were merged to create the new DMM. - What is the motivation for it, in general as well as specific terms? Today, customers choosing to adopt next-generation storage solutions have no easy method of transferring data onto these new systems while retaining access to these data by applications. A dependable, easy to use migration solution which transmits all aspects of user data greatly simplifies this task and reduces the costs associated with transition. - What are the expected benefits for Sun? Provides a migration path to allow easy adoption of new Sun storage hardware. - By what criteria will you judge its success? The project will be complete when DMM meets its goal of being able to migrate from any file system to any file system over the NFS and CIFS protocols. It will be a phased approach with interim milestones centered primarily around platforms that are supported. 2. Describe how your project changes the user experience, upon installation and during normal operation. During normal operation (i.e. no migration in progress) there will be no change to the user experience. Unless the migrator is requested to begin moving data there will be no system impact. - What does the user perceive when the system is upgraded from a previous release? Other than a potentially new or changed user interface to the migrator, there will be no perceived differences in the system. 3. What is its plan? Please see section 1.4 of the DMM Functional Specification (Program Plan Overview) - What is its current status? Has a design review been done? Are there multiple delivery phases? We completed an internal design review Feb 5-7th, 2008. We anticipate delivering DMM in multiple phases with each phase targeting different platform use cases. Phase 1 will center on migration from NAS appliances to NAS appliances using both NFS and CIFS protocols. Phase 2 will focus on migrating from generic servers to Solaris ZFS or QFS file systems. DMM is structured in a manner which allows it to be ported and hosted on non-Solaris operating systems to allow migration from any file system to any file system. While this porting work is not in-scope for DMM at the present time, it is expected that assistance from the OpenSolaris community will allow this to be brought into scope in the near future. 4. Are there related projects in Sun? See DMM Functional Specification section 1.5.4 (Projects Rendered Obsolete by this Project) and 1.6 (Competitive Analysis) - If so, what is the proposal's relationship to their work? Which not-yet- delivered Sun (or non-Sun) projects (libraries, hardware, etc.) does this project depend upon? What other projects, if any, depend on this one? DMM replaces all previous efforts within Sun for a data migration tool. - Are you updating, copying or changing functional areas maintained by other groups? How are you coordinating and communicating with them? Do they "approve" of what you propose? If not, please explain the areas of disagreement. No. 5. How is the project delivered into the system? See DMM Functional Specification section 3.1.1 (Package Overview) 6. Describe the project's hardware platform dependencies. All platforms supported by OpenSolaris for both SPARC and Intel architectures. 7. System administration - How will the project's deliverables be installed, uninstalled and (re)configured? DMM will be installed and removed with the package utilities pkgadd and pkgrm. - Does it use inetd to start itself? The DMM service is an SMF service. It will be disabled by default, and can be enabled using svcadm (1M) - Does it need installation within any global system tables? No. - Does it use a naming service such as NIS, NIS+ or LDAP? No. - What are its on-going maintenance requirements (e.g. Keeping global tables up to date, trimming files)? The DMM service will maintain a log of configuration changes and key migration events (e.g. migration start/stop). The logadm (1M) program will be used to control log file rotation. The DMM service will log migration-related errors in the syslog. The DMM service will maintain a database to store configuration and schedule information, as well as state information for each migration. Migration databases will be left intact once the migrations are complete. The majority of DMM configuration data is contained within its databases. Some elementary information will be stored in SMF (e.g. installation path, database path). - How does this project's administrative mechanisms fit into Sun's system administration strategies? E.g., how does it fit under the Solaris Management Console (SMC) and Web-Based Enterprise Management (WBEM), how does it make use of roles, authorizations and rights profiles? Additionally, how does it provide for administrative audit in support of the Solaris BSM configuration? The DMM service includes a configuration command that will be compliant with the CLIP standards. - What tunable parameters are exported? Can they be changed without rebooting the system? Examples include, but are not limited to, entries in /etc/system and ndd(8) parameters. What ranges are appropriate for each tunable? What are the commitment levels associated with each tunable (these are interfaces)? Tunable parameters specifics are currently TBD. 8. Reliability, Availability, Serviceability (RAS) - Does the project make any material improvement to RAS? N/A - How can users/administrators diagnose failures or determine operational state? (For example, how could a user tell the difference between a failure and very slow performance?) Failures resulting from a client attempting an IO will be logged and, assuming the error is not correctable by DMM, an appropriate IO error will be returned to the client. - What are the project's effects on boot time requirements? DMM will load a device driver at boot time. This driver will remain dormant until the user initiates a migration. - How does the project handle dynamic reconfiguration (DR) events? DMM's kernel module is a pseudo device driver that installs monitors within the VFS layer. DMM requires no special handling of DR events. - What mechanisms are provided for continuous availability of service? SMF will restart failed components. As necessary, user components will reset state within the kernel component. - Does the project call panic()? Explain why these panics cannot be avoided. panic() would possibly only be called in DEBUG mode. - How are significant administrative or error conditions transmitted? SNMP traps? Email notification? All error conditions are logged in both /var/adm/messages as well as DMM specific logs. SNMP traps and email notifications are not part of the scope of the project. - How does the project deal with failure and recovery? It's anticipated that the two most common failure scenarios for DMM with be complete system failure or failure to migrate data (source/target inaccessible). In the case of a complete system failure, restarting the system will initiate a failure recovery in which DMM will scan the migration database to determine what actions were in process and resume or fail those actions as appropriate. Once the database has been reconciled, the target filesystem will be mounted and exported as it would in a boot after a normal shutdown and user access can continue. In this case, the only potential for 'lost' data is for those applications that have not yet completed an outstanding I/O. In the case of a failure to migrate data -- meaning the source or target cannot be accessed for whatever reason -- the system will block client I/O to the data resolution to the issue. This could result in timeouts and failure at the protocol level for which the application logic would dictate behavior. Depending on the specific failure, administrator intervention may be required. However this failure would be no different than the failure of any storage server during client access. - Does it ever require reboot? If so, explain why this situation cannot be avoided. No, the DMM kernel component installs itself as a pseudo device driver and does not require a system reboot. - How does your project deal with network failures (including partition and re- integration)? How do you handle the failure of hardware that your project depends on? Failure of the network interface used by the client to access data will be handled by the client implementation of the protocol in question and no additional work is required for DMM to handle this failure. Failure of the network interface used by DMM to access the source storage server will result in all incoming I/O for that source data to be blocked pending resolution of the issue. It should be noted that the client will experience timeouts and eventual failure of the I/O if the issue isn't resolved within the timeouts specified by the protocol. Failure of any underlying hardware device, for example a bad disk drive on the target system which results in failure of the IO, will return the identical, or substantially similar, error to the client so that the client application may handle the error appropriately. - Can it save/restore or checkpoint and recover? The method of protecting database integrity is still being examined. Checkpoints are an alternative under consideration but the final determination is yet to be made. - Can its files be corrupted by failures? Does it clean up any locks/files after crashes? On a restart after a crash, DMM will scan it's database removing or recovering entries as appropriate. Once the recovery process completes the normal start cycle is initiated and client access is restored. 9. Observability - Does the project export status, either via observable output (e.g., netstat) or via internal data structures (kstats)? The DMM GUI will output statistical information relating to an active migration such as number of files migrated, number of bytes migrated, current directory being migrated, etc. A limited set of statistics will also be available via the CLI dmmadm command (Functional Specification 5.1 dmmadm) The DMM kernel module will make available similar statistics via kstat. - How would a user or administrator tell that this subsystem is or is not behaving as anticipated? Either through the GUI statistics screen or through CLI dmmadm command. Progress could also be observed by accessed files on the target file system. - What statistics does the subsystem export, and by what mechanism? See above. - What state information is logged? DMM maintains state information in its mySQL database. A migration log is also maintained and tracks major events. Migration-related errors are tracked via syslog. - In principle, would it be possible for a program to tune the activity of your project? DMM will have an event-based scheduling capability. Each event can be specified as a start time, a duration and a "resource utilization" number. This number will be a rough estimate as to how many system resources are to be used for that event. It's anticipated that this will be a number between 1-10. 10. What are the security implications of this project? - What security issues do you address in your project? DMM presents no new network features; it uses standard client network protocols (CIFS and NFS) to read data from source file systems located on external hosts. Access control to remote source file systems is accomplished via these protocols. Access control for local target file systems is accomplished through the privileges attached to the DMM service (see notes below). The DMM service is not enabled by default. The service will require specific authorizations for managing service state (start/stop/refresh) and changing service configuration data both in SMF as well as in the DMM database. These authorizations will be detailed in the dmm service manifest in the action authorization and value authorization properties. The dmmd service daemon must have adequate privileges to create, and delete directories, and to create, delete, and modify file data and metadata. The dmm kernel driver will check these privileges before installing migration file event monitors. The dmm service manifest will specify all privileges that the dmmd daemon is required to have in the method context of the start method. TBD: possible mount control requirements - The Solaris BSM configuration carries a Common Criteria (CC) Controlled Access Protection Profile (CAPP) -- Orange Book C2 -- and a Role Based Access Control Protection Profile (RBAC) -- rating, does the addition of your project effect this rating? E.g., does it introduce interfaces that make access or privilege decisions that are not audited, does it introduce removable media support that is not managed by the allocate subsystem, does it provide administration mechanisms that are not audited? No. Note: Auditing specifics are currently TBD. - Is system or subsystem security compromised in any way if your project's configuration files are corrupt or missing? No. - Please justify the introduction of any (all) new setuid executables. N/A - Include a thorough description of the security assumptions, capabilities and any potential risks (possible attack points) being introduced by your project. A separate Security Questionnaire http://sac.sfbay/cgi-bin/bp.cgi?NAME=Security.bp is provided for more detailed guidance on the necessary information. Cases are encouraged to fill out and include the Security questionnaire (leveraging references to existing documentation) in the case materials. Projects must highlight information for the following important areas: - What features are newly visible on the network and how are they protected from exploitation (e.g. unauthorized access, eavesdropping) - If the project makes decisions about which users, hosts, services, ... are allowed to access resources it manages, how is the requestor's identity determined and what data is used to determine if the access granted. Also how this data is protected from tampering. - What privileges beyond what a common user (e.g. 'noaccess') can perform does this project require and why those are necessary. - What parts of the project are active upon default install and how it can be turned off. See notes above. 11. What is its UNIX operational environment: - Which Solaris release(s) does it run on? OpenSolaris - Environment variables? Exit status? Signals issued? Signals caught? (See signal(3HEAD).) No environment variables are used. User commands will return with an appropriate error status. Migration daemons catch SIGINT and SIGTERM in order to perform clean up before a shutdown. - Device drivers directly used (e.g. /dev/audio)? .rc/defaults or other resource/configuration files or databases? DMM does not directly depend on any installed devices. Instead it layers on top of existing filesystems and uses client operations as one of the triggers for data migration. DMM will use a MySQL database used to track the progress of outstanding migrations. - Does it use any "hidden" (filename begins with ".") or temp files? No hidden or temporary files are used. State is maintained within DMM's database. - Does it use any locking files? No. - Command line or calling syntax: What options are supported? (please include man pages if available) Does it conform to getopt() parsing requirements? Please see the DMM Functional Specification section 5 (Command Line Interface) - Is there support for standard forms, e.g. "-display" for X programs? Are these propagated to sub-environments? DMM's CLI is a simple command line used for configuration and basic manage of DMM. It has no support for X or sub-environments. It also has no dependency on environment variables. - What shared libraries does it use? (Hint: if you have code use "ldd" and "dump -Lv")? libc.so.1 => /lib/libc.so.1 libdl.so.1 => /lib/libdl.so.1 libgen.so.1 => /lib/libgen.so.1 libintl.so.1 => /lib/libintl.so.1 libm.so.2 => /lib/libm.so.2 libmd.so.1 => /lib/libmd.so.1 libmp.so.2 => /lib/libmp.so.2 libmtmalloc.so.1 => /usr/lib/libmtmalloc.so.1 libnsl.so.1 => /lib/libnsl.so.1 libnvpair.so.1 => /lib/libnvpair.so.1 libpthread.so.1 => /lib/libpthread.so.1 librt.so.1 => /lib/librt.so.1 libscf.so.1 => /lib/libscf.so.1 libsocket.so.1 => /lib/libsocket.so.1 libsysevent.so.1 => /lib/libsysevent.so.1 libthread.so.1 => /lib/libthread.so.1 libuutil.so.1 => /lib/libuutil.so.1 - Identify and justify the requirement for any static libraries. DMM does not use static libraries. - Does it depend on kernel features not provided in your packages and not in the default kernel (e.g. Berkeley compatibility package, /usr/ccs, /usr/ucblib, optional kernel loadable modules)? No, DMM does not depend on any features not in the default kernel. - Is your project 64-bit clean/ready? If not, are there any architectural reasons why it would not work in a 64-bit environment? Does it interoperate with 64-bit versions? Yes, DMM is 64-bit ready and its ioctl interface supports 32 and 64 bit application access. - Does the project depend on particular versions of supporting software (especially Java virtual machines)? If so, do you deliver a private copy? What happens if a conflicting or incompatible version is already or subsequently installed on the system? DMM requires MySQL v5.1 or greater. DMM's installation will verify mySQL's existence at installation time and will install it if no instances are found. DMM's UI requires Java Console v3.1+, JRE v5+, Woodstock Components v4.2+ In case the of the UI dependencies, with the exception of the Woodstock Components, these components have already been integrated into OpenSolaris It's expected that the Woodstock Components will be integrated into OpenSolaris over the course of DMM development, but in the event they were not, they would be packaged with DMM. - Is the project internationalized and localized? Yes, the project will be I18N an L10N as dictated by it's target audience with available languages consistent with other Sun Solaris storage products. - Is the project compatible with IPV6 interfaces and addresses? This project does not directly access or configure the network interfaces and a prerequisite to the use of DMM will be a functional network infrastructure. As a result DMM is agnostic in regard IPV4 vs. IPV6 networking. 12. What is its window/desktop operational environment? - Is it ICCCM compliant (ICCCM is the standard protocol for interacting with window managers)? N/A. We use the web-based Woodstock component toolkit (https://woodstock.dev.java.net/index.html) - X properties: Which ones does it depend on? Which ones does it export, and what are their types? N/A. Since the GUI runs inside a browser, we do not directly depend on any X properties. - Describe your project's support for User Interface facilities including Help, Undo, Cut/Paste, Drag and Drop, Props, Find, Stop? The UI will provide online help via JavaHelp. The rest of the User Interface facilities are Woodstock compnonents dependent. - How do you respond to property change notification and ICCCM client messages (e.g. Do you respond to "save workspace")? N/A. Since the GUI resides inside a browser, it does not repond to property change notifications. - Which window-system toolkit/desktop does your project depend on? N/A. No direct dependency as long as the window-system toolkit/desktop support Mozilla firefox and Microsoft IE browsers. - Can it execute remotely? Is the user aware that the tool is executing remotely? Does it matter? The user is fully aware that the program is executed remotely. We do so by showing the current server they are working with on the top left of the display. It is important that user can manage/monitor DMM remotely. - Which X extensions does it use (e.g. SHM, DGA, Multi-Buffering? (Hint: use "xdpyinfo") N/A - How does it use colormap entries? Can you share them? N/A - Does it handle 24-bit operation? N/A 13. What interfaces does your project import and export? - Please provide a table of imported and exported interfaces, including stability levels. Pay close attention to the classification of these interfaces in the Interface Taxonomy -- e.g., "Committed," "Uncommitted," and "*Private;" see: http://sac.sfbay/cgi-bin/bp.cgi?NAME=interface_taxonomy.bp Use the following format: Interfaces Imported Interface Classification Comments The_referring_standard Committed ANSI Xy.Tz 1999 draft 37 Somebody_else () Consolidation Private Interfaces Exported Interface Classification Comments My_subroutine_name Committed MY_MACRO Project Private Etc, etc, etc... N/A - Exported public library APIs and ABIs Protocols (public or private) Drag and Drop ToolTalk Cut/Paste - Other interfaces - What other applications should it interoperate with? How will it do so? - Is it "pipeable"? How does it use stdin, stdout, stderr? DMM does not interoperate with any other applications and is not pipeable. DMM's GUI is a web based application. It will be deployed into a web container, the Sun Java Web Console 3.1. The process is done by registering our GUI to the console through a few XML files. During runtime, the Sun Java Web Console supports login and passes user credentials to the DMM GUI. - Explain the significant file formats, names, syntax, and semantics. - Is there a public namespace? (Can third parties create names in your namespace?) How is this administered? No. - Are the externally visible interfaces documented clearly enough for a non-Sun client to use them successfully? N/A 14. What are its other significant internal interfaces inter-subsystem and inter-invocation)? FEM, Solaris doors, VFS/vnode. Not a complete list. - Protocols (public or private) NFS v3/v4, CIFS - Private ToolTalk usage - Files - Other - Are the interfaces re-entrant? No. 15. Is the interface extensible? How will the interface evolve? There are no exported interfaces so there is extensibility. All interfaces are considered private and internal to the migrator. - How is versioning handled? Being part of the OpenSolaris gate, the user-space and kernel portion of DMM will implement a versioning scheme to make sure that the inbound ioctl() and outbound door calls are in sync between these two portions. Versioning will also be used between the DMM libmysql and mySQL to ensure that the database is consistent with the DMM DB API. - What was the commitment level of the previous version? N/A - Can this version co-exist with existing standards and with earlier and later versions or with alternative implementations (perhaps by other vendors)? It will not be possible to have more than one instance of DMM running per host. There should be no issues with another vendors migrator running on the same host as DMM. - What are the clients over which a change should be managed? If an upgrade is necessary then clients will lose access to the file system while the new migrator is restarted. This down time should be minimal and in no case any longer than a system reboot. - How is transition to a new version to be accomplished? What are the consequences to ISV's and their customers? New versions will remain compatible with prior versions. If any data conversion is required it will be done automatically when the new version is started. 16. How do the interfaces adapt to a changing world? - What is its relationship with (or difficulties with) multimedia? 3D desktops? Nomadic computers? Storage-less clients? A networked file system model (i.e., a network-wide file manager)? The DMM service is an application focused solely on transferring complete file system namespaces. Internal client protocol-oriented interfaces may in time need to be enhanced as protocols evolve. 17. Interoperability - If applicable, explain your project's interoperability with the other major implementations in the industry. In particular, does it interoperate with Microsoft's implementation, if one exists? N/A DMM is a stand-alone tool. - What would be different about installing your project in a heterogeneous site instead of a homogeneous one (such as Sun)? - Does your project assume that a Solaris-based system must be in control of the primary administrative node? Yes, while architected to allow a relatively easy port to something like Linux, DMM's dependency on Solaris doors and FEM regulate it at the moment to only operating on an OpenSolaris platform. 18. Performance - How will the project contribute (positively or negatively) to "system load" and "perceived performance"? There will be a slight negative impact on remote access to migrating filesystems. Users may notice slightly longer than normal delays when accessing directories or files that are being migrated. The system load should be negligible and is tunable to reduce the impact if any during normal operating hours. - What are the performance goals of the project? How were they evaluated? What is the test or reference platform? The primary goal is to avoid network timeouts when remote clients access a migrating filesystem. The performance goal will be met if the migrations delays are short enough that the client requests return successfully without timeouts. The test platforms will be Solaris (NFS and CIFS), Windows (CIFS) and Linux (NFS) clients. - Does the application pause for significant amounts of time and can the user interact with the application while it is performing long-duration tasks? No. The primary process does not pause and is always available to the user interface. Migration jobs are handled by subordinate processes which are not involved in the user interface. - What is your project's MT model? How does it use threads internally? How does it expect its client to use threads? If it uses callbacks, can the called entity create a thread and recursively call back? Internally, the processes use pthreads but this has no impact on the user. There is no requirement on clients to use or not use threads. - What is the impact on overall system performance? What is the average working set of this component? How much of this is shared/sharable by other apps? The impact on system performance will be minimal. The migrator processes are quite small and the number of these processes is tuneable. - Does this application "wake up" periodically? How often and under what conditions? What is the working set associated with this behavior? Periodic wakeups will only occur if the user has configured scheduled resource limits. Otherwise, the migrator will remain inactive until a request is sent to it by the user interface. - Will it require large files/databases (for example, new fonts)? Yes. Each migration job will maintain a mySQL database. This database can become very large depending on the size of the migrated filesystem. When the migration is complete there is no need to retain the database unless the user wishes to archive it. - Do files, databases or heap space tend to grow with time/load? What mechanisms does the user have to use to control this? What happens to performance/system load? The migration log file will tend to grow larger as a migration progresses. However, it is not expected to be excessive. A log rotation feature is supported and is available to the user if desired. 19. Please identify any issues that you would like the ARC to address. Would like guidance on where DMM components should be installed (ie, pathnames, etc) - Interface classification, deviations from standards, architectural conflicts, release constraints... - Are there issues or related projects that the ARC should advise the appropriate steering committees? 20. Appendices to include - One-Pager. - Prototype specification. - References to other documents. (Place copies in case directory.)