Disk Spanning
Disk spanning makes multiple disks appear to be one large disk. Disk spanning is frequently discussed with RAID because many software RAID solutions also provide disk spanning, but disk spanning offers no redundancy or performance benefits. It is used to create large storage systems, and some versions allow you to add disks and dynamically increase the size of the file system.
Many operating systems now include disk spanning, and in this section we cover the software solutions that come with Microsoft Windows and Linux. Other systems, such as Sun Solaris, IBM AIX, and Apple OSX, come with their own versions of disk spanning, but they are not covered here. The first part of this section covers the basic concepts and provides some general definitions that are used in all implementations. Then we cover the Windows and Linux systems. This section, like the previous RAID section, does not provide answers to every problem that you may encounter because not all the answers are known. This section describes how disk spanning works so that you can have a better understanding about why tools may not exist that meet all your needs.
Overview
The main theory behind disk spanning is similar to using a three-ring binder for taking notes instead of a spiral notebook. When you use all the pages in the spiral notebook, you must buy a new one and carry both of them around. When you use all the pages in the three-ring binder, you can add more pages to the end of it and even use a larger binder if needed. With disk spanning, the storage space from new disks is appended to the end of the existing storage space. Figure 7.6 shows an example where there are two disks being used for spanning, and each disk holds 100 data blocks. Blocks 0 to 99 are written to disk 1, and blocks 100 to 199 are written to disk 2.
Figure 7.6. Disk spanning with 100 units on the first disk and 100 units on the second disk.
A logical volume is the output of the disk-spanning software. The logical volume is made up of multiple physical disks or partitions that are sequentially merged together. Many systems also have disk groups, which are groups of physical disks, and only disks in the same group can be combined to make a logical volume. Figure 7.7 shows the relationship between the levels of abstraction. It contains three physical disks that are grouped into one disk group. The disks are spanned to form two logical volumes.
Figure 7.7. Terms and relationships for disk-spanning systems.
Linux MD
There are two methods of disk spanning in Linux. The MD driver, which we previously saw with RAID systems, also can perform basic disk spanning, and there is also the more advanced system called the Logical Volume Manager (LVM). Both systems come with major Linux distributions, and this section examines MD devices. We are covering Linux before Windows because the Linux drivers can be used to analyze a Windows system.
Background
The MD driver uses normal DOS-based partitions and groups them together to form a RAID or disk-spanning volume. Each disk can have multiple partitions and each can be used in a different RAID or logical volume. There is no notion of disk groups in this model. There is a configuration file, /etc/raidtab, that lists which partitions go in which order, and the volume cannot be mounted unless it is configured in this file. The configuration file has the same layout as was described in the previous RAID section except that the 'raid-level' setting is set to 'linear.' An example configuration file for a logical volume with two partitions (/dev/hdb1 and /dev/hdd1) is
# cat /etc/raidtab
raiddev /dev/md0
raid-level linear
nr-raid-disks 2
nr-spare-disks 0
persistent-superblock 1
chunk-size 4k
device /dev/hdb1
raid-disk 0
device /dev/hdd1
raid-disk 1
This kernel reads the configuration file and creates a device, /dev/md0, that can be mounted and used as a single volume, but it really combines the storage space of /dev/hdb1 and /dev/hdd1.
If the 'persistent' superblock value is set to 0, then the only configuration data for the device is in the /etc/raidtab file. If the value is set to 1, each disk or partition contains configuration data at the end of it that allows the auto-detect feature of Linux to automatically create the MD device. For the auto-detect process to work with Linux, the DOS partition that it is located in must have a partition type of 0xfd. You can see if the device was created during the boot by looking at the /var/log/messages log file. The auto-detect process occurs when the 'md' kernel module is loaded.
If the disks or partitions contain the persistent superblock, they have a 1024-byte structure that is broken up into sections. The first section contains settings about the resulting disk spanning or RAID volume, such as versions, the number of disks, creation time, and identifiers. The second section contains general information, such as the last update time, a counter, the state of the volume, and the number of working and failed disks. The remaining sections contain information about each of the disks in the volume, including the major and minor numbers of the device, the role of the device in the volume, and the disk's state. As we will see in the analysis section, a typical Linux system updates the values in the superblock when it boots, even if the volume is never mounted. Using the superblock, the kernel can determine if the disks have been removed from the system and placed in a different order.
Acquisition and Analysis
To analyze the data on a MD volume, your best bet is to acquire the volume as a single drive or partition and use standard analysis tools. The easiest scenario is if the volume has a persistent superblock. Otherwise, you will have to create the /etc/raidtab file on your acquisition system. If you need to create the raidtab file, then you should try to access the /etc/ directory on the original and use it as a base.
After the configuration file has been created, the MD device can be created using the raidstart command. That makes a device in the /dev/ directory based on the name given in the configuration file, and the volume can be acquired with dd or a similar tool. After the acquisition, the raidstop command needs to be executed to stop the MD device. If there is a persistent superblock, be aware that some of the values in the superblock are updated when you make a new raid device with raidstart. Ensure that you have backup images of the disks and try ATA or SCSI write blockers if you have them.
If the partition type for each of the DOS partitions in the volume is set to 0xfd, the persistent superblock exists, and the system is set to autodetect MD devices?a device will be created during startup. As with executing raidstart, this process updates the 'last update' time, the counter, and the corresponding checksum value in the superblock. This occurs even when the MD volume is not mounted!
If you place the disks into a system in a different order and location from where they originally were, the superblock will be rewritten. This happens even if the volume is not mounted, so care must be taken to reduce the changes to each disk. Furthermore, I have experienced problems when I had only one of two disks in the analysis system when I booted. The counter on that disk increased, but when I booted next with both disks, Linux would not create the MD device because they had different counter values.
I have also found that when dealing with MD disks, it is best to get into the habit of not keeping a raidtab file. It can be very dangerous because if you shut the system down and put new disks in there, Linux will try and process them as a volume. You might consider modifying your shutdown scripts to copy the raidtab file to another name so that it does not exist for the next power on.
Bootable Linux CDs can be used to acquire MD volumes, and some autodetect the volume while others do not and require that you make an /etc/raidtab file. If you can boot from a CD and create the MD device, acquire that as normal. Otherwise, acquire each disk and document which places they were in so that you minimize any modifications to the disk by putting them in a different order. I have not been able to recreate MD volumes by using a raw image of the original partitions or loopback devices. Therefore, you may need to restore the images to a disk and extract the data from the disks in the lab.
Linux LVM
The second method of disk spanning in Linux is with LVM. This section describes the design of LVM and acquisition techniques.
Background
LVM has a more advanced architecture than MD and uses the notion of disk groups, which it calls volume groups. DOS partitions that are used with LVM have a partition type of 0x8e. The disks or partitions in the volume group are divided into physical extents, which are equal-sized containers and typically have a size that is several MBs. Each system can have one or more volume groups and each volume group has a subdirectory in the /dev/ directory.
A logical volume is made up of logical extents, which have the same size as the physical extents, and there is a mapping between each logical extent and physical extent. We can see this in Figure 7.8, where a logical volume is created from three sets of physical extents from two physical disks.
Figure 7.8. LVM organizes physical disks into physical extents that are mapped to logical extents in the logical volume.
Logical volumes can be implemented by either concatenating the available physical extents or by using striping (where consecutive logical extents are on different disks). A 2GB concatenated volume may have its first 1.5GB from disk 1 and the last 500MB from disk 2. On the other hand, if the volume is striped, it may use two 1GB disks where the first 1MB is from disk 1, the second 1MB from disk 2, the third 1MB from disk 1, and so on. The logical volumes are given a device file in the volume group subdirectory in /dev/.
The configuration data for a logical volume are stored on both the local system and in the volumes. The local configuration files are stored in the /etc/lvmtab file and the /etc/lvmtab.d/ directory. The configuration files are in a binary format and are updated with the LVM utilities, such as vgimport, vgscan, and vgchange. The on-disk structure is located at the beginning of the disk and contains information about the disk, the volume group of which the disk is a member, and the logical volumes that the volume group has. None of the fields in the structure is for a time value, so the structure is not updated when the logical volume is created as occurs with MD devices.
Acquisition and Analysis
The analysis of an LVM system can be more automated than that of MD devices. The steps that I will describe here are from my experience with current versions of LVM, and I have verified them with LVM developers.[1] LVM has the vgexport and vgimport utilities that are supposed to be used when moving disks between systems, but I have found that they are not needed for acquiring the disks. The vgexport utility removes the local configuration files for the volume and writes the word "-EXPORT" in the volume group name on the disk. This step is not needed for an investigation when removing the disks from the suspect system.
[1] Email communication with Heinz Mauelshagen and A. J. Lewis. November 17, 2003.
To analyze a LVM volume, you can either remove the disks from the system and place them in a trusted Linux system or boot the suspect system from a bootable Linux CD with LVM support. As we discussed with MD, it is safest to have your systems configured to not automatically mount and configure logical volumes. When the analysis system is running, execute the vgscan command to scan the devices for logical volumes. This automatically creates the /etc/lvmtab file and configuration files in the /etc/lvmtab.d/ directory. After the configuration files have been created, the vgchange ?a y command is needed to activate the volumes that were found in the scan. With LVM, the location and master or slave configuration of the disks is not important. When the volume has become active, you can dd it from the volume device in /dev/. In my experience, using the vgscan and vgchange commands does not change the MD5 value of the disks. The command sequence can be found here where the system was booted from The Penguin Sleuth Kit (note that the Penguin Sleuth Kit is not related to The Sleuth Kit analysis tools). The Penguin Sleuth Kit is available at http://www.linux-forensics.com.
# vgscan
vgscan -- reading all physical volumes (this may take a while...)
vgscan -- found inactive volume group "vg_big2"
vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created
vgscan -- WARNING: This program does not do a VGDA backup of your volume group
# vgchange -a y
vgchange -- volume group "vg_big2" successfully activated
Note that this behavior could change in future versions of LVM, so test your procedures before you perform these actions on a real system and make disk backups before merging the volumes.
Microsoft Windows LDM
Microsoft has included support for disk spanning since Windows NT. This section will describe the design of Windows LDM and acquisition techniques.
Dynamic Disks
LDM is responsible for managing the logical volumes in Windows 2000, and XP. LDM supports simple volumes that are similar to basic partitions, disk spanning, RAID Level 0 (splitting), RAID Level 1 (mirroring), and RAID Level 5. The RAID capabilities were briefly discussed earlier in the chapter and they will be discussed in more detail here.
Basic disks are those that we saw in Chapters 5 and 6. These disks have a DOS or GPT partition table, and each partition is self-contained. Basic disks cannot be used with LDM. A dynamic disk has additional data structures to make partitions that can be used to make logical volumes. We will now discuss dynamic disks in more detail.
A dynamic disk has two important areas. The LDM partition area consumes most of the disk, and it is where dynamic partitions are created. The last 1MB of the dynamic disk is allocated to the LDM database. The LDM database contains entries to describe how the partition area is organized and how logical volumes should be created.
Each dynamic disk in an IA32 system has a DOS-based partition table in the first sector so that legacy systems will know that the disk is being used. The partition table has only one entry, and it spans the entire disk with a partition type of 0x42. The LDM partition area and database are located inside this DOS partition, as shown in Figure 7.9. We can see the partition table here using mmls from The Sleuth Kit:
# mmls ?t dos vdisk.dd
DOS Partition Table
Units are in 512-byte sectors
Slot Start End Length Description
00: ----- 0000000000 0000000000 0000000001 Primary Table (#0)
01: ----- 0000000001 0000000062 0000000062 Unallocated
02: 00:00 0000000063 0120101939 0120101877 Win LVM / Secure FS (0x42)
Figure 7.9. The layout of a dynamic disk in an IA32 system where the LDM data structures are inside a DOS partition.
Each dynamic disk in an IA64 (Intel Itanium, and so on) system has a GPT partition for the partition area and a partition for the LDM database. There are specific partition types for these partitions.
Windows supports only one disk group, so all dynamic disks are automatically assigned to it. The partition area of each dynamic disk can be partitioned into dynamic partitions. Dynamic partitions from one or more disks are grouped to make logical volumes. We can see this relationship in Figure 7.10. It is important in this section to distinguish between the terms that Microsoft uses for dynamic disks versus DOS partitions. With DOS partitions, Microsoft considers logical volumes to be partitions that are inside an extended partition, whereas with dynamic disks all partitions that can contain a file system or other data are called logical volumes.
Figure 7.10. A LDM logical volume is made up of dynamic partitions from a disk group.
The LDM Database
The LDM database is where the dynamic partitions are defined and where the rules for creating the logical volumes can be found. Microsoft has not published the exact layout and structures of the LDM database, but groups on the Internet have identified some of the internal data structures (one of the groups, Linux NTFS, is available at http://linux-ntfs.sourceforge.net). From the published Microsoft references [Soloman and Russinovich 2000], we know that the LDM database has four major sections. The Private Header is similar to the boot sector of a file system. It describes the unique characteristics of the disk and the logical volume. This structure contains a unique ID for the disk (its Windows Globally Unique Identifier (GUID)) and the name of the disk group. Windows has only one disk group, which is based on the name of your computer. The Table of Contents section is next, and its size is 16 sectors. According to Solomon and Russinovich, it "contains information regarding the databases' layout," which refers to the next section of the LDM, the database.
The database area has entries to describe disks, partitions, components, and volumes. A disk entry exists for each dynamic disk, which could be a DOS or GPT disk. The partition entries are used to describe how the dynamic disks are partitioned. The component entries are used to describe how the partitions should be merged. Each of the partition entries points to the component entry that it uses. Component entries exist for spanning, splitting, and mirroring. Lastly, the volume entries are used to describe a logical volume, which is the result of applying the component type to the partitions.
I'll give an example using two dynamic disks. We have a logical volume, and the first part of the volume is from a 15MB partition on disk 1, the second part is from a 10MB partition on disk 2, and the final part is from a 20MB partition on disk 1. Obviously, these sizes are much smaller than we would ever encounter in the real world. This is the same basic layout that was shown in Figure 7.10. The dmdiag.exe tool from Microsoft (available at http://www.microsoft.com/window2000/techinfo/reskit/tools/existing/dmdiag-o.asp) displays the entries in the database and the relevant output is as follows:
Disk: Disk1 rid=0.1027 updated=0.1122
assoc: diskid=6a565b54-b83a-4ebb-95eb-842ede926e88
flags:
Disk: Disk2 rid=0.1063 updated=0.1112
assoc: diskid=533fe4ab-0409-4ea6-98b3-9648bbc3bd12
flags:
The previous two records are the disk entries for the two physical disks. Disk1 has an ID of 0.1027, and Disk2 has an ID of 0.1063.
Group: hashDg1 rid=0.1025 update=0.1028
id: dgid=d4f40362-7794-429a-a6ad-a6dfc0553cee
diskset: id=00000000-0000-0000-0000-000000000000
copies: nconfig=all nlog=all
minors: >= 0
The previous record is a disk group entry and shows that the disk group is named with the computer name, hash.
Subdisk: Disk1-01 rid=0.1109 updated=0.1112
info: disk=0.1027 offset=0 len=30720 hidden=0
assoc: plex=0.1107 (column=0 offset=0)
Subdisk: Disk1-02 rid=0.1121 updated=0.1122
info: disk=0.1027 offset=30720 len=40960 hidden=0
assoc: plex=0.1107 (column=0 offset=51200)
The previous two entries are partition entries for the physical disk named Disk1 (ID: 0.1027). They both have a 'plex' value of 0.1107, which refers to the component entry that is used to make the logical volume. The first entry, ID 0.1109, is for the 15MB partition and has a sector offset of 0 and a length of 30,720 sectors. The second entry, ID 0.1121, is for the 20MB partition and has an offset of sector 30,720 with a length of 40,960 sectors.
Subdisk: Disk2-01 rid=0.1111 updated=0.1112
info: disk=0.1063 offset=0 len=20480 hidden=0
assoc: plex=0.1107 (column=0 offset=30720)
The previous entry is a dynamic partition entry on Disk2 (ID: 0.1063). It is a partition with ID 0.1111 with an offset of 0 sectors and a length of 20,480 sectors. We can see the relationship between the physical disk and dynamic partition entries in Figure 7.11. The direction of the arrow shows that the dynamic partition entries contain a pointer to the physical disk.
Figure 7.11. Relationship between the physical disk and dynamic partition entries in the LDM database.
Plex: Volume1-01 rid=0.1107 update=0.1124
type: layout=CONCAT
state: state=ACTIVE
assoc: vol=0.1105
The previous record is a disk-spanning (CONCAT) component entry that describes how the dynamic partitions should be combined to create a logical volume. We can see that it has an ID of 0.1107, which we saw in each of the partition entries. We also see that it is associated with Volume ID 0.1105, which is shown next.
Volume: Volume1 rid=0.1105 update=0.1124 mountname=F:
info: len=92160 guid=e40794d0-6e3c-4788-af3d-ff49d2ce769d
type: parttype=7 usetype=gen
state: state=ACTIVE
policies: read=SELECT
flags: writeback
Lastly, we see the previous record, which is for a volume entry for the logical volume. Its mounting point is F:\, and its length is 92,160 sectors. We see that it has an ID of 0.1105 and a name of 'Volume1.' The relationship of these records can be seen in Figure 7.12. Note that all disks in the disk group contain the same database entries.
Figure 7.12. Relationship between the entries in the LDM database. The arrow direction shows which objects have pointers to other objects.
We can see the final layout and organization of the logical volume in Figure 7.13. Notice that the order of the dynamic partitions is not consecutive in the logical volume.
Figure 7.13. Layout of example LDM disk with two physical disks, three dynamic partitions, and one logical volume.
The final section of the LDM database is the Transactional Log, and it is a log of modifications to the LDM. In case of power failure or crash, this is used to put the disk in a safe state.
Acquisition and Analysis
Analysis of any logical volume is difficult, especially when it is implemented in software, and it is not trivial to recreate the volume in a read-only fashion. As previously mentioned in the "RAID" section, the analysis of the system is easiest when the logical volume is acquired and standard analysis tools are used. However, this is not always possible within Windows because it attempts to mount the disks when it boots. Acquiring a mounted file system can result in a corrupt image, and the mounting may modify data. There is also a risk when moving LDM disk groups between computers. Windows supports only one disk group at a time, and any dynamic disks are added to the local group, if one exists [Microsoft 2003]. Therefore, if dynamic disks from a suspect system are imported into an analysis system with dynamic disks, the suspect disks are added to the local disk group, and this requires the OS to write new data to the LDM database.
The Linux kernel comes with support for LDM disk spanning, although it is not always enabled by default. You may need to recompile the kernel if your distribution has not enabled it. If your kernel supports LDM, Linux will read the database and create hard disk devices for each of the physical partitions in each dynamic disk. For example, if we booted Linux with the two disks from the previous example, we could have had /dev/hdb1, /dev/hdb2, and /dev/hdd1 devices. We would then have to create an /etc/raidtab file to describe the layout so that the MD kernel driver could make a single device for them. If /dev/hdb1 had the first partition, /dev/hdd1 had the second, and /dev/hdb2 had the third, the following would be the raidtab file:
raiddev /dev/md0
raid-level linear
nr-raid-disks 3
nr-spare-disks 0
persistent-superblock 0
chunk-size 4k
device /dev/hdb1
raid-disk 0
device /dev/hdd1
raid-disk 1
device /dev/hdd2
raid-disk 2
In the case of a 'linear' RAID volume, the chunk-size can be any size, but it must exist. EnCase by Guidance Software and ProDiscover from Technology Pathways can import the individual images from a Windows logical volume and merge them together.
If only disk spanning is used, you can manually extract the partitions and put them together manually from the disk images. We saw the layout details in the dmdiag.exe output, but that tool can only examine the disks from within Windows and needs to have the disks mounted. Therefore, we will use a different tool from Linux. The Linux NTFS group has developed the ldminfo tool (http://linux-ntfs.sourceforge.net/status.html#ldmtools) and it examines the LDM database entries of a Windows dynamic disk and displays them in detail when the --dump flag is given. It can be run on any of the raw devices or disk images in the volume span because all disks contain the same database entries. The output contains all the same detailed info as dmdiag.exe did, but we will only focus on the volume layout information from the example we previously used:
# ldminfo --dump disk1.dd
...
VOLUME DEFINITIONS:
Volume1 Size: 0x00016800 (45 MB)
Disk1-01 VolumeOffset: 0x00000000 Offset: 0x00000000 Length: 0x00007800
Disk2-01 VolumeOffset: 0x00007800 Offset: 0x00000000 Length: 0x00005000
Disk1-02 VolumeOffset: 0x0000C800 Offset: 0x00007800 Length: 0x0000A000
This output shows us that there are three partitions in the volume from two disks. We can easily re-create the disk because it is using spanning and not splitting. We previously saw from the output of mmls on the disk image that the partition area begins in sector 63. Therefore, we need to add 63 to the disk offset values that we see in the ldminfo output because those values are relative to the start of the partition area. The first partition is extracted by extracting 30,720 sectors (0x7800) from the partition area of the first disk:
# dd if=disk1.dd skip=63 count=30720 > span.dd
The second part of the disk span is from the first part of the second disk. We will append this data to the end of the data from the first disk. Therefore, we will extract the first 20,480 sectors (0x5000) from the partition area of the second disk:
# dd if=disk2.dd skip=63 count=20480 >> span.dd
The final part of the disk span comes from the partition area of the first disk. It will be appended to the end of what we extracted from the second disk. It starts in sector 30,720 (0x7800) of the partition area; therefore, it is sector 30,783 relative to the start of the disk, and its length is 40,960 sectors.
# dd if=disk1.dd skip=30783 count=40960 >> span.dd
We can now process the span.dd image as a normal file system image. If you have LDM support from the kernel, then I would recommend that you try that before doing it by hand. Also note that most third-party support drivers and tools for LDM have not been developed from a detailed specification from Microsoft and, therefore, may not be correct.