Digital forensics (sometimes Digital forensic science) is a branch of forensic science encompassing the recovery and investigation of material found in digital devices, often in relation to computer crime.^[1][2] The term digital forensics was originally used as a synonym for computer forensics but has expanded to cover all devices capable of storing digital data and is now used to describe the entire field.^[1] The discipline evolved in a haphazard manner during the 1990s and it was not until the early 2000s that national policies were created.

Investigations can fall into four categories. The most common category is forensic analysis, where evidence is recovered to support or oppose a hypothesis before a criminal court, this is closely related to intelligence gathering, where material is intended to identify other suspects/crimes. eDiscovery is a form of discovery related to civil litigation and intrusion investigation is a specialist investigation into the nature and extent of an unauthorized network intrusion. The technical side of investigations is divided into several sub-branches; computer forensics, network forensics, database forensics and mobile device forensics. Any number of the fields may be utilised in an investigation.

As well as identifying direct evidence of a crime, digital forensics can be used to attribute evidence to specific suspects, confirm alibis or statements, determine intent, identify sources (for example, in copyright cases) or authenticate documents.^[3] Investigations are much broader in scope than other areas of forensic analysis (where the usual aim is to provide answers to a series of simpler questions) often involving complex time-lines or hypothesis.^[4]

The digital forensic process encompasses the seizure, forensic imaging (acquisition) and analysis of digital media. Finally producing a report of the digital evidence for the courts or an employer. Computer devices tend to store large amounts of information in cache/log files and deleted space and forensic examiners can recover this data as part of the analysis process.

History

Aerial photo of FLETC, where US digital forensics standards were developed in the 80s and 90s

Prior to the 1980s crimes involving computers were dealt with using existing laws. The first computer crimes were recognized in the 1978 Florida Computer Crimes Act, which included legislation against the unauthorized modification or deletion of data on a computer system.^[5][6] Over the next few years the range of computer crimes being committed increased, and laws were passed to deal with issues of copyright, privacy/harassment (e.g., cyber bullying, cyber stalking, and online predators) and child pornography.^[7][8] It was not until the 1980s that federal laws began to incorporate computer offences. Canada was the first country to pass legislation in 1983.^[6] This was followed by the US Federal Computer Fraud and Abuse Act in 1986, Australian amendments to their crimes acts in 1989 and the British Computer Abuse Act in 1990.^[6][8]

1980s-1990s: Growth of the field

The growth in computer crime during the 1980s and 1990s caused law enforcement agencies to begin establishing specialized groups, usually at the national level, to handle the technical aspects of investigations. For example, in 1984 the FBI launched a Computer Analysis and Response Team and the following year a computer crime department was set up within the British Metropolitan Police fraud squad. Besides being law enforcement professionals many of the early members of these groups were also computer hobbyists and became responsible for the fields initial research and direction.^[9][10]

One of the first practical (or at least publicised) examples of digital forensics was Cliff Stoll's pursuit of hacker Markus Hess in 1986. Stoll, whose investigation made use of both computer and network forensic techniques, was not a specialised examiner.^[11] Many of the earliest forensic examinations followed the same profile.^[12]

Throughout the 1990s there was high demand for the these new, and basic, investigative resources. The strain on central units lead to the creation of regional, and even local, level groups to help handle the load. For example, the British National Hi-Tech Crime Unit was set up in 2001 to provide a national infrastructure for computer crime; with personnel located both centrally in London and with the various regional police forces (the unit was folded into the Serious Organised Crime Agency (SOCA) in 2006).^[10]

During this period the science of digital forensics grew out of ad-hoc tools and techniques developed by these hobbyist practitioners. This is in contrast to other forensics disciplines which developed from work by the scientific community.^[1][13] It was not until 1992 that the term "computer forensics" was used in academic literature (although prior to this it had been in informal use); a paper by Collier and Spaul attempted to justify this new discipline to the forensic science world.^[14][15] This swift development resulted in a lack of standardization and training. In his 1995 book, "High-Technology Crime: Investigating Cases Involving Computers", K Rosenblatt wrote:^[6]

Seizing, preserving, and analyzing evidence stored on a computer is the greatest forensic challenge facing law enforcement in the 1990s. Although most forensic tests, such as fingerprinting and DNA testing, are performed by specially trained experts the task of collecting and analyzing computer evidence is often assigned to patrol officers and detectives^[16]

2000s: Developing standards

Since 2000, in response to the need for standardization, various bodies and agencies have published guidelines for digital forensics. The Scientific Working Group on Digital Evidence (SWGDE) produced a 2002 paper, "Best practices for Computer Forensics", this was followed, in 2005, by the publication of an ISO standard (ISO 17025, General requirements for the competence of testing and calibration laboratories).^[17][6][18] A European lead international treaty, the Convention on Cybercrime, came into force in 2004 with the aim of reconciling national computer crime laws, investigative techniques and international co-operation. The treaty has been signed by 43 nations (including the US, Canada, Japan, South Africa, UK and other European nations) and ratified by 16.

The issue of training also received attention. Commercial companies (often forensic software developers) began to offer certification programs and digital forensic analysis was included as a topic at the UK specialist investigator training facility, Centrex.^[6][10]

Since the late 1990s mobile devices have become more widely available, advancing beyond simple communication devices, and have been found to be be rich forms of information, even for crime not traditionally associated with digital forensics.^[19] Despite this, digital analysis of phones has lagged behind traditional computer media, largely due to problems over the proprietary nature of devices.^[20]

Focus has also shifted onto internet crime, particularly the risk of cyber warfare and cyberterrorism. A February 2010 report by the U.S. Joint Forces Command concluded:

Through cyberspace, enemies will target industry, academia, government, as well as the military in the air, land, maritime, and space domains. In much the same way that airpower transformed the battlefield of World War II, cyberspace has fractured the physical barriers that shield a nation from attacks on its commerce and communication.^[21]

The field of digital forensics still faces outstanding issues. A 2009 paper, "Digital Forensic Research: The Good, the Bad and the Unaddressed", by Peterson and Shenoi identified a bias towards Windows operating systems in digital forensics research.^[22] In 2010 Simson Garfinkel identified issues facing digital investigations in the future; including the increasing size of digital media, the wide availability of encryption to consumers, a growing variety of operating systems and file formats, an increasing number of individuals owning multiple devices and legal limitations on investigators. The paper also identified continued training issues, as well as the prohibitively high cost of entering the field.^[11]

Investigative tools

During the 1980s very few specialised digital forensic tools existed and investigators often performed live analysis on media, examining computers from within the operating system using existing sysadmin tools to extract evidence. This risked modifying data on the disk (inadvertently or otherwise) leading to claims of evidence tampering. In the early 1990s a number of tools were created to allow investigations to take place without the risk of altering data.

The need for such software was first recognised in 1989 at the Federal Law Enforcement Training Center, resulting in the creation of IMDUMP (by Michael White) and in 1990, SafeBack (developed by Sydex). Similar pieces of software were produced in other countries; DIBS (a hardware and software solution) was released commercially in the UK in 1991 and Rob McKemmish released Fixed Disk Image free to Australian law enforcement.^[9] These tools allowed examiners to create an exact copy of a piece of digital media to work on; leaving the original disk intact for verification. By the end of the 90s, as demand for digital evidence grew more advanced, commercial tools (EnCase, FTK, etc.) were developed that allowed analysts to examine copies of media without using any live forensics.^[6]

More recently the same progression of tool development has occurred for mobile devices; initially investigators accessed data directly on the device, these were soon replaced with specialist tools (such as XRY or Radio Tactics Aceso).^[6]

Forensic Process

A portable Tableau write-blocker attached to a Hard Drive

A digital forensic investigation commonly consists of 3 stages; acquisition or imaging of exhibits, analysis and reporting.^[6][23] Acquisition involves creating an exact sector level duplicate (or "forensic duplicate") of the media, often using a write blocking device to prevent modification of the original. Both acquired image and original media are hashed (using SHA-1 or MD5) and the values compared to verify the copy is accurate.^[24]

During the analysis phase an investigators recover evidence material using a number of different methodologies and tools. In 2002 and article in the International Journal of Digital Evidence referred to this step as "an in-depth systematic search of evidence related to the suspected crime".^[1] In 2006, forensics researcher Brian Carrie described an "intuitive procedure" in which obvious evidence is first identified and then "exhaustive searches are conducted to start filling in the holes."^[4]

The actual process of analysis can vary between investigations, but common methodologies include conducting keyword searches across the digital media (within files as well as unallocated and slack space), recovering deleted files and extraction of registry information (for example to list user accounts, or attached USB devices).

Once evidence is recovered it is analysed to reconstruct events or actions and to reach conclusions, work that can often be performed by less specialist staff.^[1] When an investigation is complete the data is presented, usually in the form of a written report, in lay persons terms.^[1]

Forms and uses

An example of an image's Exif metadata that might be used to prove its origin

Investigations can take one of four forms. Firstly as a forensic examination, traditionally associated with criminal law, where evidence is collected to support or oppose a hypothesis. Like other areas of forensics this is often part of a wider investigation spanning a number of disciplines. A related form is "intelligence gathering", functionally identical to a forensic examination the digital evidence is intended to be used as intelligence to (for example) locate, identify or halt other crimes. As a result intelligence gathering is sometimes held to a less strict forensic standard.

In civil litigation or corporate matters the process is referred to as electronic discovery (or eDiscovery). The forensic procedure is similar to that used in criminal investigations but with different legal requirements and limitations. Finally, intrusion investigation is a specialist examination into the nature and extent of an unauthorized network intrusion. Intrusion analysis is usually performed as a damage limitation exercise after an attack, both to establish the extent of any intrusion and to try and identify the attacker.^[4][3] Such attacks were commonly conducted over phone lines during the 1980s, but in the modern era are usually propagated over the internet.^[25]

The main use of digital forensics is to recover objective evidence of a criminal activity (termed actus reus in legal parlance). However the diverse range of data held in digital devices can help with other areas of investigation.^[3]

Attribution

Meta data and other logs can be used to attribute actions to an individual. For example, personal documents on a computer drive might identify its owner.

Alibis and statements

Information provided by those involved can be cross checked with digital evidence. For example, during the investigation into the Soham murders, the offenders alibi was disproven when mobile phone records of the person he claimed to be with showed she was out of town at the time.

Intent

As well as finding objective evidence of a crime being committed, investigations can also be used to prove the intent (known by the legal term mens rea). For example, the Internet history of convicted killer Neil Entwistle included references to a site discussing How to kill people.

Evaluation of source

File artifacts and meta-data can be used to identify the origin of a particular piece of data; for example, older versions of Microsoft Word embedded a Global Unique Identifer into files which identified the computer it had been created on. Proving whether a file was produced on the digital device being examined or obtained from elsewhere (e.g., the Internet) can be very important.^[3]

Document authentication

Related to "Evaluation of Source", meta data associated with digital documents can be easily modified (for example, by changing the computer clock you can affect the creation date of a file). Document authentication relates to detecting and identifying falsification of such details.

Limitations

One major limitation to a forensic investigation is the use of encryption; this disrupts initial examination where pertinent evidence might be located using keywords. Laws to compel individuals to disclose encryption keys are still relatively new and controversial.^[11]

Legal considerations

The examination of digital media is covered by national and international legislation. For civil investigations, in particular, laws may restrict the abilities of analysts to undertake examinations. Restrictions against network monitoring, or reading of personal communications often exist.^[26] For criminal investigation, national laws restrict how much information can be seized.^[26] For example, in the United Kingdom seizure of evidence by law enforcement is governed by the PACE act.^[6] The "International Organization on Computer Evidence" (IOCE) is one agency that works to establish compatible international standards for the seizure of evidence.^[27]

In the UK the same laws covering computer crime can also affect forensic investigators. The 1990 computer misuse act legislates against unauthorised access to computer material; this is a particular concern for civil investigators who have more limitations than law enforcement.

An individuals right to privacy is one area of digital forensics which is still largely undecided by courts. The US Electronic Communications Privacy Act places limitations on the ability of law enforcement or civil investigators to intercept and access evidence. The act makes a distinction between stored communication (e.g. email archives) and transmitted communication (such as VOIP). The latter, being considered more of a privacy invasion, is harder to obtain a warrant for.^[6][16] The ECPA also affects the ability of companies to investigate the computers and communications of their employees, an aspect that is still under debate as to the extent to which a company can perform such monitoring.^[6]

In Europe Article 5 of the European Convention on Human Rights asserts similar privacy limitations to the ECPA. In addition it also limits the processing and sharing of personal data both within the EU and with external countries. In the UK the ability of law enforcement to conduct digital forensics investigations is legislated by the Regulation of Investigatory Powers Act.^[6]

Digital evidence

Digital evidence can come in a number of forms

Where it will be used in a court of law, digital evidence falls under the same legal guidelines as other forms of evidence, courts do not usually require more stringent guidelines.^[6][28] In the United States the Federal Rules of Evidence are used to evaluate the admissibility of digital evidence, the United Kingdom PACE and Civil Evidence acts have similar guidelines and many other countries have their own laws. US federal laws restrict seizures to items with only obvious evidential value. This is acknowledged as not always being possible to establish with digital media prior to an examination.^[26]

Laws dealing with digital evidence refer to two considerations; integrity and authenticity. Integrity is ensuring that the act of seizing and acquiring digital media does not modify the evidence (either the original or the copy). Authenticity refers to the ability to confirm the integrity of information; for example that the imaged media matches the original evidence.^[26] The ease with which digital media can be modified means that documenting the chain of custody from the crime scene, through analysis and, ultimately, to the court, (a form of audit trail) is important to establish the authenticity of evidence.^[6]

Attorneys have argued that because digital evidence can theoretically be altered it undermines the reliability of the evidence. US judges are beginning to reject this theory, in the case US v. Bonallo the court ruled that "the fact that it is possible to alter data contained in a computer is plainly insufficient to establish untrustworthiness".^[6][29] In the United Kingdom guidelines such as those issued by ACPO are followed to help document the authenticity and integrity of evidence.

Digital investigators, particularly in criminal investigations, have to ensure that conclusions are based upon factual evidence and their own expert knowledge.^[6] In the US, for example, Federal Rules of Evidence state that a qualified expert may testify “in the form of an opinion or otherwise” so long as:

(1) the testimony is based upon sufficient facts or data, (2) the testimony is the product of reliable principles and methods, and (3) the witness has applied the principles and methods reliably to the facts of the case.^[30]

Many of the sub-branches of digital forensics have their own specific guidelines for handling and investigating evidence. For example, mobile phones are often acquired inside a Faraday shield to stop radio traffic to the device. Or, in the UK forensic examination of computers in criminal matters is subject to ACPO guidelines.^[6]

Branches

Digital forensics includes several sub-branches relating to the investigation of various types of devices, media or artefacts.

Computer forensics

The goal of computer forensics is to explain the current state of a digital artifact; such as a computer system, storage medium or electronic document.^[31] The discipline usually covers computers, embedded systems (digital devices with rudimentary computing power and onboard memory) and static memory (such as USB pen drives).

Computer forensics can deal with a broad range of information; from logs (such as internet history) through to the actual files on the drive. In 2007 prosecutors used a spreadsheet recovered from the computer of Joseph E. Duncan III to show premeditation and secure the death penalty.^[3] Sharon Lopatka's killer was identified in 2006 after email messages from him detailing torture and death fantasies were found on her computer.^[6]

Mobile phones in a UK Evidence bag

Mobile device forensics

Mobile device forensics is a sub-branch of digital forensics relating to recovery of digital evidence or data from a mobile device. It differs from Computer forensics in that a mobile device will have an inbuilt communication system (e.g. GSM) and, usually, proprietary storage mechanisms. Investigations usually focus on simple data such as call data and communications (SMS/Email) rather than in-depth recovery of deleted data.^[6][32] SMS data from a mobile device investigation helped to exonerate Patrick Lumumba in the murder of Meredith Kercher.^[3]

Mobile devices are also useful for providing location information; either from inbuilt gps/location tracking or via cell site logs (which track the devices within their range). Such information was used to track down the kidnappers of Thomas Onofri in 2006.^[3]

Network forensics

Network forensics relates to the monitoring and analysis of computer network (both local network and WAN/internet) traffic for the purposes of information gathering, legal evidence or intrusion detection.^[33] Traffic is intercepted (usually at the packet level) and either stored for later analysis with specialist tools or filtered in real time for relevant information. Unlike other areas of digital forensics, network data is often volatile and seldom logged, making the discipline often reactionary.

In 2000, the FBI lured computer hackers Aleksey Ivanov and Gorshkov to the United States for a fake job interview. By monitoring network traffic from the pair's computers, the FBI identified passwords that let it collect evidence directly from Russia-based computers.^[6][34]

Database forensics

Database forensics is a branch of digital forensics relating to the forensic study of databases and their metadata.^[35] Investigations use database contents, log files and in-RAM data in order to build a time-line or recover relevant information.

See also

• Cyberspace
• Glossary of digital forensics terms

Related journals

• Journal of Digital Forensics, Security and Law
• International Journal of Digital Crime and Forensics
• Journal of Digital Investigation
• International Journal of Digital Evidence
• International Journal of Forensic Computer Science
• Journal of Digital Forensic Practice
• Small Scale Digital Device Forensic Journal

References

Further reading

Look up digital forensics in Wiktionary, the free dictionary.

Wikibooks has more on the topic of Digital forensics

• Carrier, Brian D. (February 2006). "Risks of live digital forensic analysis". Communications of the ACM 49 (2): 56–61. doi:10.1145/1113034.1113069. ISSN 0001-0782. http://portal.acm.org/citation.cfm?id=1113034.1113069. Retrieved 31 August 2010. 
• Kanellis, Panagiotis. Digital crime and forensic science in cyberspace. IGI Publishing. p. 357. ISBN 1591408733. http://books.google.co.uk/books?id=oK_oYhTPW2gC. 
• Jones, Andrew (2008). Building a Digital Forensic Laboratory. Butterworth-Heinemann. p. 312. ISBN 1856175103. http://books.google.co.uk/books?id=F5IU7XXKwCQC. 
• Marshell, Angus M. (2008). Digital forensics: digital evidence in criminal investigation. Wiley-Blackwell. p. 148. ISBN 0470517751. http://books.google.co.uk/books?id=MC0FPQAACAAJ. 
• "Timeline of computer forensics". http://www.pc-history.org/forensics.htm.

HFS Plus or HFS+ is a file system developed by Apple Inc. to replace their Hierarchical File System (HFS) as the primary file system used in Macintosh computers (or other systems running Mac OS). It is also one of the formats used by the iPod digital music player. HFS Plus is also referred to as Mac OS Extended (or, erroneously, “HFS Extended”), where its predecessor, HFS, is also referred to as Mac OS Standard (or, erroneously, as “HFS Standard”). During development, Apple referred to this filesystem with the codename Sequoia.^[3]

HFS Plus is an improved version of HFS, supporting much larger files (block addresses are 32-bit length instead of 16-bit) and using Unicode (instead of Mac OS Roman or any of several other character sets) for naming the items (files, folders) – names which are also character encoded in UTF-16^{[verification needed]} and normalized to a form very nearly the same as Unicode Normalization Form D (NFD)^[4] (which means that precomposed characters like are decomposed in the HFS+ filename and therefore count as two characters^[5] and UTF-16 implies that characters from outside the Basic Multilingual Plane — often seldom used and characters from ancient writing systems — also count as two characters in an HFS+ filename). HFS Plus permits filenames up to 255 UTF-16 characters in length, and n-forked files similar to NTFS, though until recently, almost no software takes advantage of forks other than the data fork and resource fork. HFS Plus also uses a full 32-bit allocation mapping table, rather than HFS’s 16 bits. This was a serious limitation of HFS, meaning that no disk could support more than 65,536 allocation blocks under HFS. When disks were small, this was of little consequence, but as larger-capacity drives became available, it meant that the smallest amount of space that any file could occupy (a single allocation block) became excessively large, wasting significant amounts of space. For example, on a 1 GB disk, the allocation block size under HFS is 16 KB, so even a 1 byte file would take up 16 KB of disk space. Unlike most other file systems HFS Plus supports hard links to directories.

Like HFS, HFS Plus uses B-trees to store most volume metadata.

History

HFS+ was introduced with the January 19, 1998 release of Mac OS 8.1.^[1] However its first appearance, as a beta filesystem, was in the never-released Copland OS betas.

With the release of the 10.2.2 update on November 11, 2002, Apple added optional journaling features to HFS Plus for improved data reliability. These features were easily accessible in Mac OS X Server, but only accessible through the command line in the standard desktop client.^[6] With Mac OS X v10.3, all HFS Plus volumes on all Macs are set to be journaled by default. Within the system, an HFS Plus volume with a journal is identified as HFSJ.

10.3 also introduced another version of HFS Plus called HFSX. HFSX volumes are almost identical to HFS Plus volumes, except that they are never surrounded by the HFS Wrapper that is typical of HFS Plus volumes and they optionally support case sensitivity for file and folder names. HFSX volumes can be recognized by two entries in the Volume Header, a value of HX in the signature field and 5 in the version field.^[1]

Additionally, Mac OS X 10.3 marked Apple's adoption of Unicode 3.2 decomposition, superseding the Unicode 2.1 decomposition used previously. This change has caused problems for developers writing software for Mac OS X.^[7]

With 10.4, Apple added support for Inline Attribute Data records, something that had been a part of the Mac OS X implementation of HFS Plus since at least 10.0, but always marked as "reserved for future use".^[8] Until the release of Mac OS X Server 10.4, HFS Plus supported only the standard UNIX file system permissions, however 10.4 introduced support for access control list-based file security, which provides a richer mechanism to define file permissions and is also designed to be fully compatible with the file permission models on other platforms such as Microsoft Windows XP and Windows Server 2003.^[9]

Design

HFS Plus volumes are divided into sectors (called logical blocks in HFS), that are usually 512 bytes in size. These sectors are then grouped together into allocation blocks which can contain one or more sectors; the number of allocation blocks depends on the total size of the volume. HFS Plus uses a larger value to address allocation blocks than HFS, 32 bits rather than 16 bits; this means it can access 4,294,967,296 (= 2³²) allocation blocks rather than the 65,536 (= 2¹⁶) allocation blocks available to HFS.^[1]

Formerly, HFS Plus volumes were embedded inside an HFS standard filesystem. This was phased out by the Tiger transition to Intel Macs, where the HFS Plus filesystem was not embedded inside a wrapper. The wrapper was designed for two purposes; it allowed Macintosh computers without HFS Plus support in their ROM to boot HFS Plus volumes and it also was designed to help users transition to HFS Plus by including a minimal HFS volume with a read-only file called Where_have_all_my_files_gone?, explaining to users with versions of Mac OS 8.0 and earlier without HFS Plus, that the volume requires a system with HFS Plus support. The original HFS volume contains a signature and an offset to the embedded HFS Plus volume within its volume header. All allocation blocks in the HFS volume which contain the embedded volume are mapped out of the HFS allocation file as bad blocks.^[1]

There are nine structures that make up a typical HFS Plus volume:^[1]

1. Sectors 0 and 1 of the volume are HFS boot blocks. These are identical to the boot blocks in an HFS volume. They are part of the HFS wrapper.
2. Sector 2 contains the Volume Header equivalent to the Master Directory Block in an HFS volume. The Volume Header stores a wide variety of data about the volume itself, for example the size of allocation blocks, a timestamp that indicates when the volume was created or the location of other volume structures such as the Catalog File or Extent Overflow File. The Volume Header is always located in the same place.
3. The Allocation File which keeps track of which allocation blocks are free and which are in use. It is similar to the Volume Bitmap in HFS, in which each allocation block is represented by one bit. A zero means the block is free and a one means the block is in use. The main difference with the HFS Volume Bitmap, is that the Allocation File is stored as a regular file, it does not occupy a special reserved space near the beginning of the volume. The Allocation File can also change size and does not have to be stored contiguously within a volume.
4. The Catalog File is a B-tree that contains records for all the files and directories stored in the volume. The HFS Plus Catalog File is very similar to the HFS Catalog File, the main differences being records are larger to allow more fields and to allow for those fields to be larger (for example to allow the longer 255-character unicode file names in HFS Plus). A record in the HFS Catalog File is 512 bytes in size, a record in the HFS Plus Catalog File is 4 KB in Mac OS and 8 KB in Mac OS X. Fields in HFS are of fixed size, in HFS Plus the size can vary depending on the actual size of the data they store.
5. The Extents Overflow File is another B-tree that records the allocation blocks that are allocated to each file as extents. Each file record in the Catalog File is capable of recording eight extents for each fork of a file; once those are used extents are recorded in the Extents Overflow File. Bad blocks are also recorded as extents in the Extents Overflow File. The default size of an extent record in Mac OS is 1 KB and 4 KB in Mac OS X.
6. The Attributes File is a new B-tree in HFS Plus that does not have a corresponding structure in HFS. The Attributes File can store three different types of 4 KB records: Inline Data Attribute records, Fork Data Attribute records and Extension Attribute records. Inline Data Attribute records store small attributes that can fit within the record itself. Fork Data Attribute records contain references to a maximum of eight extents that can hold larger attributes. Extension Attributes are used to extend a Fork Data Attribute record when its eight extent records are already used.
7. The Startup File is designed for non-Mac OS systems that don't have HFS or HFS Plus support. It is similar to the Boot Blocks of an HFS volume.
8. The second to last sector contains the Alternate Volume Header equivalent to the Alternate Master Directory Block of HFS.
9. The last sector in the volume is reserved for use by Apple. It is used during the computer manufacturing process.

Other operating systems

Linux

The Linux kernel includes the hfsplus module^[10] for mounting HFS+ filesystems. HFS+ fsck and mkfs have been ported to Linux and are part of the hfsprogs package.^[11] These drivers currently have issues with corruption of HFS+ drives with a capacity greater than 2 TB. As such Linux distributions such as Ubuntu do not allow mounting of HFS+ drives or partitions greater than 2 TB.^[12]

The Linux HFS+ kernel driver has support to read and write to HFS+ non-journaled drives/parititions but only has read support of journaled HFS+. Journaling ability was added to HFSplus when Mac OS X came out and is by default on for Mac OS X installations. Journaling is a redundant behavior of a filesystem that helps protect data loss. If planning to write to an HFS+ partition then drive journaling must be turned off in OSX.^[13]

Windows

On Windows, a fairly complete filesystem driver for HFS+ exists as a commercial software package called MacDrive.^[14] This package allows Windows users to read and write HFS+ formatted drives, and read Mac-format optical disks.

Another solution is provided by Paragon, with their HFS+ for Windows driver; this supports both read and write on HFS+ partitions.^[15]

Apple has released read-only HFS+ drivers for Windows XP, Windows Vista, and Windows 7 in Boot Camp in Mac OS X 10.6. Microsoft has created a HFS+ driver for the XBox 360 mainly for the purpose of reading HFS+ formatted iPods.

A free (GPL) alternative to MacDrive is HFSExplorer written by Erik Larsson.^[16] HFSExplorer is an application for viewing and extracting files from an HFS+ volume (Mac OS Extended) or an HFSX volume (Mac OS Extended, Case-sensitive) located either on a physical disk, on a .dmg disk image, or in a raw file system dump. HFSExplorer is less complete than MacDrive in the sense that it can read, but not write to HFS formatted volumes.

See also

• Comparison of file systems

References

External links

• Apple Technote 1150 - HFS Plus Volume Format
• Apple Technote 1189 - The Monster Disk Driver Technote
• hfsdebug - A debugger for HFS Plus volumes by Amit Singh
• hfsprogs - Userspace support tools for HFS+ filesystems under Linux systems (adapted from Apple's native tools).
• iBored - A disk editor and viewer supporting HFS Plus

The ext2 or second extended filesystem is a file system for the Linux kernel. It was initially designed by Rémy Card as a replacement for the extended file system (ext).

The canonical implementation of ext2 is the ext2fs filesystem driver in the Linux kernel. Other implementations (of varying quality and completeness) exist in GNU Hurd, MINIX 3, Mac OS X (third-party), Darwin (same third-party as Mac OS X but untested), some BSD kernels, in Atari MiNT, and as third-party Microsoft Windows drivers.

ext2 was the default filesystem in several Linux distributions, including Debian and Red Hat Linux, until supplanted more recently by ext3, which is almost completely compatible with ext2 and is a journaling file system. ext2 is still the filesystem of choice for flash-based storage media (such as SD cards, and USB flash drives) since its lack of a journal minimizes the number of writes and flash devices have only a limited number of write cycles. Recent kernels, however, support a journal-less mode of ext4, which would offer the same benefit along with a number of ext4-specific benefits.

History

The early development of the Linux kernel was made as a cross-development under the Minix operating system. Naturally, it was obvious that the Minix file system would be used as Linux's first file system. The Minix file system was mostly free of bugs, but used 16-bit offsets internally and thus only had a maximum size limit of 64 megabytes. There was also a filename length limit of 14 characters. Because of these limitations, work began on a replacement native file system for Linux.

To ease the addition of new file systems and provide a generic file API, VFS, a virtual file system layer was added to the Linux kernel. The extended file system (ext), was released in April 1992 as the first file system using the VFS API and was included in Linux version 0.96c. The ext file system solved the two major problems in the Minix file system (maximum partition size and filename length limitation to 14 characters), and allowed 2 gigabytes of data and filenames of up to 255 characters. But it still had problems: there was no support for separate access, inode modification and data modification timestamps.

As a solution for these problems, two new filesystems were developed in January 1993: xiafs and the second extended file system (ext2), which was an overhaul of the extended file system incorporating many ideas from the Berkeley Fast File System. ext2 was also designed with extensibility in mind, with space left in many of its on-disk data structures for use by future versions.

Since then, ext2 has been a testbed for many of the new extensions to the VFS API. Features such as POSIX ACLs and extended attributes were generally implemented first on ext2 because it was relatively simple to extend and its internals were well-understood.

On Linux kernels prior to 2.6.17,^[1] restrictions in the block driver mean that ext2 filesystems have a maximum file size of 2TB.

ext2 is still recommended over journaling file systems on bootable USB flash drives and other solid-state drives. ext2 performs fewer writes than ext3 since it does not need to write to the journal. As the major aging factor of a flash chip is the number of erase cycles, and as those happen frequently on writes, this increases the life span of the solid-state device.^[2] Another good practice for filesystems on flash devices is the use of the noatime mount option, for the same reason.

ext2 data structures

The space in ext2 is split up into blocks. These blocks are divided into block groups, analogous to cylinder groups in the Unix File System. There are typically thousands of blocks on a large file system. Data for any given file is typically contained within a single block group where possible. This is done to reduce external fragmentation and minimize the number of disk seeks when reading a large amount of consecutive data.

Each block group contains a copy of the superblock and block group descriptor table, and all block groups contain a block bitmap, an inode bitmap, an inode table and finally the actual data blocks.

The superblock contains important information that is crucial to the booting of the operating system, thus backup copies are made in multiple block groups in the file system. However, typically only the first copy of it, which is found at the first block of the file system, is used in the booting.

The group descriptor stores the location of the block bitmap, inode bitmap and the start of the inode table for every block group and these, in turn are stored in a group descriptor table.

Inodes

Every file or directory is represented by an inode. The inode includes data about the size, permission, ownership, and location on disk of the file or directory.

Example of ext2 inode structure:

Quote from the linux kernel documentation for ext2:

"There are pointers to the first 12 blocks which contain the file's data in the inode. There is a pointer to an indirect block (which contains pointers to the next set of blocks), a pointer to a doubly-indirect block (which contains pointers to indirect blocks) and a pointer to a trebly-indirect block (which contains pointers to doubly-indirect blocks)."

So, there is a structure in ext2 that has 15 pointers, the first 12 are for direct blocks. Pointer number 13 points to an indirect block, number 14 to a doubly-indirect block and number 15 to a trebly-indirect block.

Directories

Each directory is a list of directory entries. Each directory entry associates one file name with one inode number, and consists of the inode number, the length of the file name, and the actual text of the file name. To find a file, the directory is searched front-to-back for the associated filename. For reasonable directory sizes, this is fine. But for huge large directories this is inefficient, and ext3 offers a second way of storing directories that is more efficient than just a list of filenames.

The root directory is always stored in inode number two, so that the file system code can find it at mount time. Subdirectories are implemented by storing the name of the subdirectory in the name field, and the inode number of the subdirectory in the inode field. Hard links are implemented by storing the same inode number with more than one file name. Accessing the file by either name results in the same inode number, and therefore the same data.

The special directories "." and ".." are implemented by storing the names "." and ".." in the directory, and the inode number of the current and parent directories in the inode field. The only special treatment these two entries receive is that they are automatically created when any new directory is made, and they cannot be deleted.

Allocating Data

When a new file or directory is created, the EXT2 file system must decide where to store the data. If the disk is mostly empty, then data can be stored almost anywhere. However, performance is maximized if the data is clustered with other related data to minimize seek times.

The EXT2 file system attempts to allocate each new directory in the group containing its parent directory, on the theory that accesses to parent and children directories are likely to be closely related. The EXT2 file system also attempts to place files in the same group as their directory entries, because directory accesses often lead to file accesses. However, if the group is full, then the new file or new directory is placed in some other non-full group.

The data blocks needed to store directories and files can found by looking in the data allocation bitmap. Any needed space in the inode table can be found by looking in the inode allocation bitmap.

File system limits

The reason for some limits of the ext2-file system are the file format of the data and the operating system's kernel. Mostly these factors will be determined once when the file system is built. They depend on the block size and the ratio of the number of blocks and inodes. In Linux the block size is limited by the architecture page size.

There are also some userspace programs that can't handle files larger than 2 GB.

The maximum file size is limited to min( ((b/4)³+(b/4)²+b/4+12)*b, 2³²*b ) due to the i_block (an array of EXT2_N_BLOCKS) and i_blocks( 32-bits integer value ) representing the amount of b-bytes "blocks" in the file.

The limit of sublevel-directories is 31998 due to the link count limit. Directory indexing is not available in ext2, so there are performance issues for directories with a large number of files (10,000+). The theoretical limit on the number of files in a directory is 1.3 × 10²⁰, although this is not relevant for practical situations.

Note: In Linux kernel 2.4 and earlier block devices were limited to 2 TB, limiting the maximum size of a partition regardless of block size.

Compression extension

e2compr is a modification to the ext2 file system driver in the Linux kernel to support online compression and decompression of files on file system level without any support by user applications.

e2compr is a small patch against the ext2 file system that allows on-the-fly compression and decompression. It compresses only regular files; the administrative data (superblock, inodes, directory files etc.) are not compressed (mainly for safety reasons). Access to compressed blocks is provided for read and write operations. The compression algorithm and cluster size is specified on a per-file basis. Directories can also be marked for compression, in which case every newly created file in the directory will be automatically compressed with the same cluster size and the same algorithm that was specified for the directory.

e2compr is not a new file system. It is only a patch to the ext2 file system made to support the EXT2_COMPR_FL flag. It does not require you to make a new partition, and will continue to read or write existing ext2 file systems. One can consider it as simply a way for the read and write routines to access files that could have been created by a simple utility similar to gzip or compress. Compressed and uncompressed files coexist nicely on ext2 partitions.

The latest e2compr-branch is available for current releases of 2.6 and 2.4 Linux kernels, but development is stalled. There are also older branches for older 2.0 and 2.2 kernels, which are more stable.

See also

• e2fsprogs
• ext : ancestor of ext2
• ext3 : extended version of ext2
• ext4 : fourth extended file system
• StegFS : a steganographic file system based on ext2
• cloop
• SquashFS
• List of file systems
• Comparison of file systems
• Filesystem in Userspace (FUSE)

References

Notes

Further reading

• John Newbigin. John's spec of the second extended filesystem. http://uranus.chrysocome.net/explore2fs/es2fs.htm. 
• Dave Poirier (2009). The Second Extended File System: Internal Layout. http://www.nongnu.org/ext2-doc/index.html. 
• Theodore Ts'o and Stephen Tweedie (June, 2002). "Planned Extensions to the Linux Ext2/Ext3 Filesystem". USENIX 2002 Annual Technical Conference. http://e2fsprogs.sourceforge.net/extensions-ext23/. 
• Larry Ayers (June 1997). "E2compr: Transparent File Compression for Linux". Linux Gazette (18). http://linuxgazette.net/issue18/e2compr.html. 
• Charles Cong and Jeremy H. Brown (1997-12-26) (PDF). A System for Transparent File Compression With Caching Under Linux. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.51.4780&rep=rep1&type=pdf. 
• Charles Cong and Jeremy H. Brown (1997-10-22) (PDF). A Survey of Modern File Compression Techniques. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.50.9847&rep=rep1&type=pdf. 

External links

• ext2fs user-space tools
• Read only software for ext2, ext3 on Windows
• Ext2read A windows application to read/copy ext2/ext3/ext4 files with extent and LVM2 support.
• A Free ext2 driver for Windows NT4.0/2000/XP/2003/Vista/2008 with full access to Linux Ext2 volumes (read access and write access). (Read-only access for ext3, too - see the FAQ)(Freeware)
• Ext2Fsd GPL ext2/ext3 file system driver for Windows 2000/XP/2003/VISTA/2008 (opensource, supports read & write, works with FreeOTFE)

• ext2에서 자료 삭제 및 손실 없이 ext3으로 변경할 수 있다(자료를 백업할 필요가 없음).
• 저널링
• 온라인 파일 시스템 증대
• 큰 규모의 디렉터리를 위한 Htree(btree의 고급판)

파일 시스템(file system, 파일체계)은 컴퓨터에서 파일이나 자료를 쉽게 발견 및 접근할 수 있도록 보관 또는 조직하는 체제를 가리키는 말이다.

파일 시스템은 통상 하드 디스크나 CD-ROM 같은 실제 자료 보관 장치를 사용하여 파일의 물리적 소재를 관리하는 것을 가리키나 네트워크 프로토콜(NFS, SMB, 9P 등)을 수행하는 클라이언트를 통하여 파일 서버 상의 자료로의 접근을 제공하는 방식과 가상의 형태로서 접근 수단만이 존재하는 방식(procfs 등)도 파일 시스템의 범위에 포함될 수 있다. 디렉터리 서비스나 레지스트리와는 의미가 조금 다르다.

구성

하드 디스크 내장 / 파티션 모형

개요

파일 시스템은 일반적으로 크기가 일정한 블록들의 배열(섹터라고도 불리며 통상 512바이트, 1키비바이트, 2키비바이트같은 - 2를 제곱한 수만큼의 크기를 갖는다)에 접근할 수 있는 자료 보관 장치 위에 생성되어 이러한 배열들을 조직함으로 파일이나 디렉터리를 만들며 어느 부분이 파일이고 어느 부분이 공백인지를 구분하기 위하여 각 배열에 표시를 해 둔다. 또한 자료를 '클러스터' 또는 '블록'이라고 불리는 일정한 단위(이것은 각 디스크 배열들에 대한 식별할 수 있는 번호를 제공하는데 통상 1부터 64까지가 쓰인다)에 새겨 넣는데 이것이 바로 파일 하나가 필요로 하는 디스크의 최소 공간이다.

파일 이름

파일에 이름이 존재함은 컴퓨터 메모리 안에서 해당 파일의 소재 위치를 보장하기 위함이다. 파일 시스템 안에서 디렉터리는 통상 파일 이름과 해당 파일을 연결해 주는데 도스의 FAT나 유닉스 계열 운영 체제의 아이노드의 경우 파일 이름을 색인과 연결한다. 디렉터리 구조는 수평형일 수도 있고 수직형(하위 디렉터리 있음)일 수도 있다. 어떤 파일 시스템에서는 파일 이름이 이름, 확장자, 판 수 이런 식으로 특별한 문법을 따르기도 하며 또 어떤 파일 시스템에서는 파일 이름은 그저 문자열 정도로만 취급되고 각 파일마다 어딘가에 메타데이터가 보관된다.

계층

일반적으로 파일 시스템은 '저장 장치 - 입출력 제어 - 기본 파일시스템 - 파일조직 모듈 - 가상 파일시스템'과 같이 여러 개의 계층으로 구성되어 있다. 그렇게 함으로써 하드 디스크, 시디롬 드라이브, 플래시 메모리 등 다양한 형태의 저장 장치를 지원할 수 있고, 하나의 시스템에 여러 개의 파일 시스템을 사용하는 것이 가능해진다.

파일 시스템의 종류

파일 시스템은 크게 디스크, 네트워크, 그리고 특수 용도의 파일 시스템으로 나눌 수 있다.

디스크 파일 시스템

자료 기억 장치, 특히 컴퓨터에 연결된 디스크 드라이브에 파일을 저장하도록 설계된 시스템이다.

보기

• 파일 할당 테이블
• NTFS
• HFS
• ext2
• ext3
• ext4
• ISO 9660
• ODS-5
• UDF
• 유닉스 파일 시스템
• ZFS

데이터베이스 파일 시스템

데이터베이스 기반의 파일 시스템은 최근에 등장한 새로운 개념의 파일 시스템이다. 파일을 계층 구조로 관리하지 않고 파일의 형식, 주제, 만든이, 내용과 같은 여러 특성에 따라 시스템에서 자동으로 분류하여 관리하는 것이다. 따라서 쿼리 언어나 자연어 등으로 파일을 빠르게 찾을 수 있다.

보기

• Gnome VFS
• BFS
• WinFS

트랜잭션 기반 파일 시스템

트랜잭션 기반 파일 시스템은 파일에 일어난 이벤트나 트랜잭션을 기록하는 시스템이다. 사용자가 수행하는 작업은 여러 개의 파일의 내용을 바꿀 수 있다. 이 바뀐 내용들은 서로 연관되어 있는 경우가 많으므로 이 내용들이 논리적으로 서로 연결되어 있어야 하는 시스템에서는 이 변화들이 동시에 일어난다는 것이 보장되어야 한다. 은행 계좌에서 돈을 이체하는 경우를 예로 들어 보자. 당신이 온라인으로 돈을 이체할 때 은행 컴퓨터는 상대방의 은행 컴퓨터에 "전송" 명령을 보내고 동시에 당신의 계좌에서 같은 금액을 줄일 것이다. 그런데 이때 시스템에 우연히 사고가 일어나 전송 명령이 보내지지 않았다면 상대방의 계좌에 돈은 입금되지 않았는데 당신의 계좌에서는 돈이 사라지는 일이 일어날 수 있다. 트랜잭션 기반 시스템은 이렇게 논리적으로 동시에 수행되어야 하는 작업들을 하나의 "트랜잭션"으로 묶어 만약의 사고가 일어났을 때 양쪽에서 트랜잭션을 다시 수행하여 오류를 막는다. 또한 모든 트랜잭션은 기록으로 남아 어디서 무슨 일이 언제 수행되었는지가 기록된다. 이러한 파일 시스템은 시스템의 오류를 막기 위해 설계되었으며, 느리지만 안전하다고 볼 수 있다.

특수 용도의 파일 시스템

유닉스와 같은 파일 중심의 운영 체제는 여러 가지 특수 용도의 파일 시스템을 사용한다. 예를 들면, 어떤 종류의 유닉스는 '/proc'이라는 파일 시스템에서 프로세스나 운영 체제 여러 기능에 접근할 수 있다.

심우주 탐사선인 보이저 1호와 2호에는 디지털 테이프 기반의 파일 시스템이 탑재되어 있다. 카시니-하위헌스 호와 같은 현대 우주선들은 실시간 운영 체제를 위한 파일 시스템을 탑재한다. 화성 탐사선들도 실시간 운영 체제를 탑재하고 있는데, 이들의 파일 시스템은 플래시 메모리를 사용한다.

파일 시스템과 운영 체제

대부분의 운영 체제는 파일 시스템을 갖고 있으며, 파일은 현대의 모든 운영 체제의 기본 구성 요소이다. 마이크로컴퓨터의 초창기 운영 체제 도스(곧, 디스크 운영 체제)의 주 목적은 파일 관리였다. 이러한 초창기 운영 체제들은 디스크를 관리하는 시스템을 따로 설계했다. 이러한 운영 체제들은 내부에 단 하나의 파일 시스템만을 지원했다.

유닉스 계열의 파일 시스템

유닉스나 다른 유닉스 계열 운영 체제들은 여러 개의 주변 장치에 각각의 이름을 붙이지만, 그 주변 장치에 존재하는 파일들은 전부 하나의 계층 구조 아래 관리된다. 다시 말하면, 유닉스에서는 하나의 루트 디렉터리가 있고, 운영 체제에서 접근할 수 있는 모든 파일들은 전부 루트 디렉터리 아래의 어느 디렉터리에 들어 있다. 또한, 루트 디렉터리는 어떤 특정한 하드 디스크에 존재할 필요가 없고, 심지어 네트워크 위의 가상 파일 공간을 루트 디렉터리로 삼을 수도 있다.

다른 주변 장치에 있는 파일에 접근하려면, 이 주변 장치의 파일 시스템을 어떤 디렉터리로 놓을 것인지를 운영 체제에게 알려야 한다. 이것을 가리켜 "파일 시스템을 마운트한다"고 일컫는다. 예를 들어, 유닉스에서 시디롬을 읽으려면 시디롬의 파일 시스템을 루트 밑의 /mnt 디렉터리 밑에 나타나게 하는 명령을 내린다. 일반적으로 컴퓨터의 관리자만이 파일 시스템을 마운트할 수 있다.

리눅스의 파일 시스템

• Extended File System (ext , ext2 , ext3 , ext4)
• ZFS
• ResierFS
• XFS

솔라리스의 파일 시스템

• ZFS

맥 OSX 10의 파일 시스템

맥 OSX10은 맥OS에서 쓰던 HFS를 개량한 HFS Plus를 사용한다. 많은 메타데이터를 가지며 대소문자를 구분하지 않는다. HFS와 달리 유닉스식 접근 권한 관리 기능이 있으며 나중에는 저널링과 함께 파일 시스템 단편화 관련 여러 알고리즘이 더해졌다.

파일 이름은 255자까지 지을 수 있고 파일 이름을 저장할 때 유니코드를 쓴다. 맥 오에스 텐에서 파일 형식은 파일 이름의 확장자로 알 수도 있고 메타데이터에 저장된 타입 코드로부터 알 수도 있다.

HFS Plus는 다음의 세 가지 링크를 사용한다.

• 하드 링크
• 심볼릭 링크
• 에일리어스(alias): 파일이 옮겨지거나 이름이 바뀌어도 링크가 깨지지 않는다.

마이크로소프트 윈도의 파일 시스템

마이크로소프트 윈도는 초창기 운영 체제(CP/M-80을 바탕으로 만든 MS-DOS) 기반으로 제작되었지만, 다른 운영 체제(유닉스, OS/2 등)의 파일 시스템과 사용자 인터페이스에서 많은 아이디어를 빌려왔다. 윈도는 FAT와 NTFS를 사용한다. 초창기 FAT 파일 시스템은 파일 이름의 길이에 제한이 있었고, 디스크와 파티션 수에도 마찬가지로 제한이 있었다.

윈도 NT에 탑재되어 함께 출시된 NTFS는 접근 제어 리스트 기반의 권한 설정과 하드 링크, 여러 개의 파일 스트림, 쿼터 추적, 압축, 다른 파일 시스템을 마운트하는 기능 등이 있다.

다른 운영 체제들과는 달리, 드라이브 글자를 사용하여 디스크나 파티션을 구분한다. 예를 들어, C:WINDOWS는 글자 C로 보여지는 파티션 안에 있는 WINDOWS라는 디렉터리임을 뜻한다. C 드라이브는 운영 체제가 설치된 첫 번째 하드 디스크 파티션을 나타내는 글자로 많이 쓰인다. 이것은 MS-DOS 시절, A와 B는 플로피 디스크 드라이브를 가리키고, C 드라이브가 하드 디스크를 가리켰기 때문이다. 이런 전통 때문에, 운영 체제가 설치된 파티션이 C 드라이브라고 가정하는 오래된 프로그램들이 버그를 일으키곤 한다. 네트워크 드라이브 또한 드라이브 글자로 매핑될 수 있다.