Certification Test Matrix

Certification Test Matrix

Goals:

• Ensure compliance with virtual appliance best practices
• Ensure compatibility across VMware product line
• Ensure ease-of-use for initial customer experience with appliance

	Interpret base requirement exactly as written
	Interpret base requirement exactly the opposite as written (MUST vs. MUST NOT, SHOULD vs. SHOULD NOT)
	This test case is not applicable to this target

Test Matrix:

Test Case Series 01xx [Platforms] TC ID TC Title TC Summary Hypervisor Class Hosted Class OVF 0110 Required platform targets. The following platforms MUST be targeted: Hypervisor vHW 4 -- ESXi 3.5.x;Hypervisor vHW 4 -- ESX 3.5.x; 0120 Recommended platform targets. The following platforms SHOULD be targeted: Hosted vHW 4 -- Linux Host OS;Hosted vHW 4 -- Win32 Host OS; 0130 Highly-portable platform targets. For additional portability options, the following platforms MAY be targeted: OVF Revision 0.9; 0140 Domain-specific optional platform targets. For certain domain-specific applications, the following platforms MAY be targeted: Hosted vHW 6 -- Linux Host OS;Hosted vHW 6 -- Mac OS X Host OS;Hosted vHW 6 -- Win32 Host OS; Test Case Series 05xx [Packaging] TC ID TC Title TC Summary Hypervisor Class Hosted Class OVF 0505 No Extraneous Files. The virtual appliance archive SHOULD NOT contain extraneous files. An example of such files would be vmware.log (or equivalent) outputs. 0510 Packaging Complexity. The use of multiple archives to encapsulate a single virtual appliance may be confusing to the end user. If multiple archives are used, extraction expectations MUST be included to explain that the net result is a single directory with the contents of all archives in it (net-count ~"N" files). Some archive extraction tools may balk at performing an overlay-extraction, the Microsoft Windows XP built-in "Extraction Wizard" being a good example when handling .ZIP archives. 0511 Packaging with .tar (Tape ARchive). Use of the .tar archive format for packaging MAY be implemented at the vendor's discretion, and so-stipulated in the Documentation (see Test Cases 1001 through 1099). Note that certain limitations exist in the in the .tar archive format regarding original file sizes. 0512 Packaging with .tar.gz/.tgz (Tape ARchive GZipped). Use of the .tar.gz/.tgz archive and compression format for packaging MAY be implemented at the vendor's discretion, and so-stipulated in the Documentation (see Test Cases 1001 through 1099). Note that certain limitations exist in the in the .tar archive format regarding original file sizes. 0513 Packaging with .tar.bz2/.tbz2 (Tape ARchive BZip2ed). Use of the .tar.bz2/.tbz2 archive and compression format for packaging MAY be implemented at the vendor's discretion, and so-stipulated in the Documentation (see Test Cases 1001 through 1099). Note that certain limitations exist in the in the .tar archive format regarding original file sizes. 0514 Packaging with .tar.7z/.t7z (Tape ARchive 7Zipped). Use of the .tar.7z/.t7z archive and compression format for packaging MAY be implemented at the vendor's discretion, and so-stipulated in the Documentation (see Test Cases 1001 through 1099). Note that certain limitations exist in the in the .tar archive format regarding original file sizes. 0515 Packaging with .zip (Zipped). Use of the .zip archive and compression format for packaging MAY be implemented at the vendor's discretion, and so-stipulated in the Documentation (see Test Cases 1001 through 1099). 0520 Message Digest Hashes for Integrity Validation. In order to allow for integrity validation of archives, message digesting algorithms such as MD5 and SHA1 SHOULD be used. Test Case Series 10xx [Documentation] TC ID TC Title TC Summary Hypervisor Class Hosted Class OVF 1001 Availability of Documentation. Documentation MUST be available as Part of the Virtual Appliance. 1002 Cursory Overview. A Cursory Overview SHOULD be available for review by a potential customer prior to acquisition of the appliance. This SHOULD be limited to one (1) page or less, and SHOULD include a mix of non-technical (deployment scenarios) and technical (time and resource estimates) information. 1003 Getting Started Guide. A Getting Started Guide MUST be available as Part of the Virtual Appliance. This document SHOULD be limited to ten (10) pages or less, and SHOULD include: Installation Procedures, Pre-Boot Configuration Procedures, and First-Boot Configuration Procedures. 1011 Delivery Formats. Documentation MUST be delivered in a portable and user-friendly format. Primary format options: Plaintext (.txt) -- UTF-8(?);Adobe Acrobat Portable Document Format (.pdf) -- Minimum Version(?) and Maximum Version(?);Offline XHTML or HTML (.html) -- XHTML 1.0 Transitional recommended;Online XHTML or HTML (.html) -- XHTML 1.0 Transitional recommended; Other secondary format options (RTF, LaTeX, compiled MS Help, ...) 1020 General Content Requirements. At this time, core documentation MUST be delivered in English. Any variance to this policy MUST be made through written arrangement. 1021 Editing and Proofreading. Documentation MUST be free of major editing and proofreading (spelling, grammar, punctuation) issues. 1031 Accuracy of Procedures. All procedural information MUST be correct. 1041 Branding and Terminology. All documentation MUST communicate coherent branding to the end customer. It is understood that companies and/or products undergo periodic name changes. The intent of this requirement is that such naming be consistent, either always reflecting the "new" name, or always reflecting the "old" name. 1051 Branding and Terminology (VMware and third party). All documentation MUST include accurate branding and terminology for both VMware and third-parties. It is understood that part of maintaining brand identity is ensuring consistent use of that brand, which includes requiring that the full name and proper capitalization/punctuation be used in references. 1052 Branding and Terminology (VMware and third party). All documentation MUST include registered trademark / trademark / service mark / trade dress disclaimer. Boilerplate verbiage such as the following used by VMware will suffice: All other marks and names mentioned herein may be trademarks of their respective companies. 1061 Resource Requirements and Prerequisites. Documentation MUST include an accurate listing of resource requirements and other prerequisites (CPU/RAM/Disk/Network) for deployment of the virtual appliance. 1062 External Resource Requirements and Prerequisites. Documentation MUST include an accurate listing of external resource requirements and other prerequisites for deployment of the virtual appliance, including but not limited to: A description of each virtual network interface so that mappings to the appropriate virtual switch(es) are made during deployment.;A description of other virtual appliances this virtual appliance is dependent on.;A description of other virtual appliances dependent on this virtual appliance.;A description of services (rDNS, DNS, DHCP, NAT/PAT, NTP, etc...) this virtual appliance is dependent on.;A description of static network configuration data this virtual appliance is dependent on.; 1063 External Resource References. Documentation SHOULD encourage the use of fully qualified hostnames rather than non-qualified "short" hostnames when accessing network resources. Use of "short" hostnames makes a potentially incorrect assumption regarding the local client's DNS stub-resolver configuration, specifically the domain-search list and other auto-completion behavior for non-qualified hostnames. 1065 Performance Expectations. Documentation SHOULD include some description of performance expectations contrasted to resource requirements, such as: Does adding more RAM help the VA?;Does adding a second vCPU help the VA?;Does adding additional spindles to the logical disk backing the VMFS3 volume help the VA?;Is there a minimum sustained storage IOPS rate required from the logical disk?; Test Case Series 25xx [Virtual Resources] TC ID TC Title TC Summary Hypervisor Class Hosted Class OVF 2501 Virtual Resources -- Prohibitions. The use of the following resources are explicitly prohibited at this time: Virtual Disk, IDE Sub-Type -- MUST NOT be present; 2502 Virtual Resources -- Restrictions. The use of the following resources are strongly discouraged at this time: Virtual USB, USB 1.1 (original) Sub-Type -- SHOULD NOT be present;Virtual USB, USB 2.0 (EHCI) Sub-Type -- SHOULD NOT be present;Virtual Legacy Device, Floppy Sub-Type -- SHOULD NOT be present;Virtual Legacy Device, IDE CD-ROM Sub-Type -- SHOULD NOT be present;Virtual Legacy Device, Serial Port Sub-Type -- SHOULD NOT be present;Virtual Legacy Device, Parallel Port Sub-Type -- SHOULD NOT be present; 2509 Virtual Resources -- Platform Generation. The migration to newer virtual hardware (vHW) versions is strongly encouraged by VMware, however early adoption may limit the target platforms available to the virtual appliance. For broadest applicability and unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), a virtual hardware (vHW) version of vHW 4 SHOULD be used. In addition, unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), a virtual hardware (vHW) version earlier than vHW 3 or later than vHW 6 MUST NOT be present and/or required for proper operation of the virtual appliance. 2510 Virtual CPU -- General. By nature, VMware vPlatform products perform virtualization of the Intel IA-32 (aka 'x86') PC architecture, including "64 bit extensions" (aka 'x86_64'). This means that a large percentage of the OpCodes (instructions at the 'assembly' level) submitted by the Guest OS to the virtual CPU are run without modification directly on the physical CPU (aka direct execution, or 'DirectExec'). We only fully-emulate or perform binary-translation on those specific OpCodes or series of OpCodes which would present an inherent virtualization risk if run in direct execution mode. The virtual appliance MUST NOT require other hardware architectures (such as SPARC/uSPARC, PA-RISC, IA-64/Itanium, ARM, MIPS) for proper operation. 2511 Virtual CPU -- More than Zero vCPUs. At least one vCPU MUST be present and/or required for proper operation of the virtual appliance. -- Yes, this seems logically redundant, but may be useful for machine-processing of this specification. 2512 Virtual CPU -- Two vCPUs. Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), two virtual CPUs (vCPUs) SHOULD NOT be present and/or required for proper operation of the virtual appliance. 2513 Virtual CPU -- More than Two vCPUs. Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), more than two virtual CPUs (vCPUs) MUST NOT be present and/or required for proper operation of the virtual appliance. 2515 Virtual CPU -- AMD NX (No-Execute) / Intel xD (Execute-Disable) Bit. Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), hardware support (pCPU level) for the AMD NX and/or Intel xD bit MUST NOT be required for proper operation of the virtual appliance. 2516 Virtual CPU -- CPUID Bit Masking (Hide / Expose). Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), boot-time and privileged-mode CPUID bit masking MUST NOT be present and/or required for proper operation of the virtual appliance. -- CPUID 0FxT1 filters 2517 Virtual CPU -- CPUID Bit Forcing (Present / Not-Present). Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), boot-time and privileged-mode CPUID bit forcing MUST NOT be present and/or required for proper operation of the virtual appliance. -- CPUID RFxTH filters 2519 Virtual CPU -- Features Not Otherwise Specified. Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), hardware support (pCPU level) for specialty features MUST NOT be present and/or required for proper operation of the virtual appliance. -- CPUID spoofing, etc... 2520 Virtual RAM -- General. Allocated virtual RAM MUST be a multiple of 4 MB. 2521 Virtual RAM -- Minimum. Allocated virtual RAM MUST be a minimum of 32 MB. 2522 Virtual RAM -- 3.5 GB Limit. Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), allocated virtual RAM MUST NOT exceed 3600 MB. Note that for certain platform targets this is a hard limit, in which case allocated virtual RAM MUST NOT exceed 3600 MB. 2523 Virtual RAM -- 8 GB Limit. Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), allocated virtual RAM MUST NOT exceed 8192 MB. 2524 Virtual RAM -- 16 GB Limit. Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), allocated virtual RAM MUST NOT exceed 16384 MB. 2529 Virtual RAM -- Maximum Limit. Allocated virtual RAM MUST NOT exceed 65536 MB. 2530 Virtual Storage Controller -- General. SCSI required. LSI Logic strongly preferred over BusLogic. 2540 Virtual Disk -- General. Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), all requisite virtual disks MUST be present for proper operation of the virtual appliance. 2541 Virtual Disk -- Types. All virtual disks MUST be SCSI. 2542 Virtual Disk -- Backing Types -- Plain. All virtual disks included as Part of the Virtual ApplianceMUST be backed by plain virtual disks. 2543 Virtual Disk -- Backing Types -- Other. Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), all virtual disks MUST be backed by plain virtual disks. Other backing types (raw LUN, RDM, physical disk passthrough, etc...) require explicit documentation. 2550 Virtual Networking Controller -- General. Where possible, the vmxnet virtual NIC SHOULD be used. Certain guests may require the use of the e1000 virtual NIC or vlance virtual NIC. 2560 Virtualization Acceleration Devices -- General. The prudent use of virtualization acceleration devices is strongly encouraged by VMware, however such use may limit the target platforms available to the virtual appliance. Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), virtualization acceleration devices other than those described elsewhere MUST NOT be present and/or required for proper operation of the virtual appliance. 2561 Virtualization Acceleration Devices -- VMI. Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), the VMI device (aka paravirt interface) MUST NOT be present and/or required for proper operation of the virtual appliance. 2562 Virtualization Acceleration Devices -- VMCI. Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), the VMCI device (Virtual Machine Communication Interface) MUST NOT be present and/or required for proper operation of the virtual appliance. 2563 Virtualization Acceleration Devices -- NPIV. Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), the Fibre-Channel NPIV/VPORT device (N-Port ID Virtualization - VMkernel Virtual Port) MUST NOT be present and/or required for proper operation of the virtual appliance. 2564 Virtualization Acceleration Devices -- Virtual MMU. Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), the Virtual MMU device (Hardware Page Table Virtualization) MUST NOT be present and/or required for proper operation of the virtual appliance. 2570 VMware Tools -- General. The core set of Guest OS side functionality components known as "VMware Tools" MUST be present, enabled, and operating properly. Information further defining this requirement is covered in test cases 2571 through 2579. 2571 VMware Tools -- Host-Guest Communications. The subset of Guest OS side functionality components known as the "guest daemon" MUST be present, enabled, and operating properly. 2572 VMware Tools -- Guest to Host Log Channel. Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), the subset of Guest OS side functionality components known as the "guest to host log channel" MUST be present, enabled, and operating properly. 2573 VMware Tools -- Memory Balloon Driver. Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), the subset of Guest OS side functionality components known as the "memory balloon driver" (vmmemctl) MUST be present, enabled, and operating properly. 2574 VMware Tools -- Not Otherwise Specified. Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), the subset of Guest OS side functionality components known as "Host-Guest Filesystem" (vmhgfs) MUST NOT be implemented. 2579 VMware Tools -- Not Otherwise Specified. The subset of Guest OS side functionality components other than those described elsewhere MAY be implemented at the vendor's discretion, and so-stipulated in the Documentation (see Test Cases 1001 through 1099). 2580 Virtual Legacy Devices -- General. Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), virtual legacy devices MUST NOT be present and/or required for proper operation of the virtual appliance. 2589 Virtual Resource -- Not Otherwise Specified. Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), resources other than those described elsewhere MUST NOT be present and/or required for proper operation of the virtual appliance. Test Case Series 30xx [VMware .vmx File] TC ID TC Title TC Summary Hypervisor Class Hosted Class OVF 3001 Annotation option in .vmx file. The annotation option SHOULD be set in the .vmx file, this option shows up as a free-form single-line wrapped text field in some but not all user interfaces, typically as "Notes" in the bottom-left corner of the UI. Typically it is set as a more verbose version of the displayName option, including for example the company's URL. Annotation values can span up to a paragraph or so, but must be stored in the .vmx file as a single "long" line of text. 3050 File Paths -- General. File paths MUST be local (bar.baz), and not relative (../foo/bar.baz) or absolute (/foo/bar.baz). 3051 File Paths -- NVRAM. nvram = "...." 3052 File Paths -- Log. log.fileName = "...." 3053 File Paths -- Virtual Disk (SCSI). scsi#:#.fileName = "...." 3054 File Paths -- Virtual Disk (IDE). ide#:#.fileName = "...." 3055 File Paths -- Extended Configuration. extendedConfigFile = "...." 3056 File Paths -- working Directory. workingDir = "." Test Case Series 79xx [Guest OS Selection] TC ID TC Title TC Summary Hypervisor Class Hosted Class OVF 7901 Guest OS is Supported. The Guest Operating System selected for use in the virtual appliance MUST be one which is listed as Supported for the target platform. The current descriptions of Supported Guest OS variants can be found in HTML at http://pubs.vmware.com/guestnotes/ and in PDF at http://www.vmware.com/pdf/GuestOS_guide.pdf (revised concurrently). 7902 Guest OS is 32 bit. Unless otherwise stipulated in the Documentation (see Test Cases 1001 through 1099), the Guest Operating System selected for use in the virtual appliance MUST be 32 bit. Test Case Series 80xx [Guest OS and App. Stack] TC ID TC Title TC Summary Hypervisor Class Hosted Class OVF 8001 Filesystem consistency. All Guest OS filesystems MUST be marked "clean". No filesystem presented to the Guest OS may be in a state where consistency checking (fsck / chkdsk / etc...) is required as part of the normal deployment process of a virtual appliance. Written exceptions MAY be granted for situations where the filesystem is, by design, intended to be in an inconsistent state, for example a computer forensics training environment. 8002 Filesystem mount-count and age data. All Guest OS filesystems SHOULD have their mount-count and age values reset to extend the period of time before the first forcible consistency check (fsck / chkdsk / etc...) is required as part of the normal operation of a virtual appliance. 8003 Missing and remnant filesystems. With the exception of properly documented resources added as part of the deployment process, all Guest OS required filesystems MUST be present. Stale mount-point entries (CD-ROMs, NAS shares, etc...) and other remnants of pre-deployment work MUST NOT be present. 8011 Authentication and cryptography components. Authentication and cryptographic keying components SHOULD NOT be present in the Guest OS and/or application stack. Presence of these components means that every iteration of the virtual appliance in the field will have precisely the same keying (ex: SSH and/or SSL) components. The ISV is responsible for determining relevance, whether or not this is an appropriate behavior, and MUST validate that the virtual appliance is capable of properly regenerating keying components (e.g. automatically running keygen tool) should they be removed.