ext3

장점

이 밖의 모든 것들은 ext2와 같다. ext2를 유지하고 복구하기 위해 충분한 테스트를 거쳐 보다 완전해진 파일 시스템 유지보수 유틸리티들을 포함하여 ext2 파일 시스템에서 큰 변화 없이 ext3와 함께 사용될 수 있도록 하였다. ext2와 ext3 둘 다 e2fsprogs를 사용하며 이 유틸리티는 fsck를 포함하고 있다. 이러한 밀접한 관련으로 이 두 파일 시스템들은 상호 변환이 용이하다.

저널링

Ext3를 지원하는 리눅스 시스템에서는 다음과 같은 3단계 저널링을 사용할 수 있다.

• Journal (리스크 최소)

두 파일 시스템의 메타 데이터와 파일 컨텐츠는 메인 파일 시스템에 전달되기 전에 저널에 기록된다. 저널은 비교적 디스크와 관련이 있어서 어떤 경우에는 성능을 향상시킬 수 있으나, 데이터가 저널에 한 번, 파일 시스템에 한 번, 이렇게 두 번 기록되기 때문에 성능이 저하될 수도 있다.

• Ordered (리스크 중간)

메타 데이터만 저널에 기록된다. 파일 컨텐츠는 기록되지는 않지만 만일 관련된 메타 데이터가 저널에 기록되면 파일 컨텐츠는 디스크에 반드시 기록된다. 이는 많은 리눅스 배포판에 기본 설정으로 되어 있다. 만일 파일을 읽거나 쓰는 도중에 전원이 갑자기 꺼지거나 커널 패닉 상태가 되면, 저널은 새로운 파일을 가리키게 되거나 추가된 데이터가 넘겨지지 않으며, 삭제 처리된다. 하지만, 중복 쓰기가 된 파일은 원본이 저장되지 않아 파일이 손상될 수 있는데, 파일을 복구하기 위한 충분한 정보 없이 새 파일과 이전 파일의 중간 상태에서 파일이 종료될 수 있다. - 새로운 데이터는 완벽하게 디스크에 저장되지 않으며, 이전 데이터는 어디에도 저장되지 않는다. - 심한 경우에는, 중간 상태가 이전 데이터와 새 데이터 사이에 혼란을 줄 수 있다.^[1][2]

• Writeback (리스크 최고)

메타 데이터만 저널에 기록되며, 파일의 내용은 기록되지 않는다. 파일 내용은 저널이 업데이트된 후에나 아니면 그 이전에 기록될 수 있으며, 결과적으로 충돌 바로 전에 수정된 파일들은 손상될 수 있다. 예를 들어, 추가된 파일이 실제 크기보다 더 큰 파일로 저널에 기록되면, 결국은 "쓰레기(의미 없는 정보)"를 만들게 된다. 오래된 파일일수록 저널이 복구된 후에 예상치 못한 결과가 나타날 수 있다. 데이터와 저널 사이에 동시성이 결여되며 대부분의 경우에서 점점 심해진다. XFS와 JFS는 이러한 저널링 레벨을 사용하지만 데이터를 기록하지 않기 때문에 모든 "쓰레기"는 재부팅 시 완전히 삭제된다.

일부 상황에서는 동적 inode 할당 및 확장과 같은 현대 파일시스템의 기능 부족이 단점으로 여겨질 수 있지만, 복구의 측면에서는 이러한 사실이 아주 뛰어난 장점이 된다. 파일 시스템의 메타 데이터는 모두 수정되고, 잘 알려진 위치에 존재하며, 데이터 구조에 일부 중복성이 내재되어 있어, 트리 기반의 파일 시스템이 복구되기 어려운 상황에서도 뚜렷한 데이터 손상에도 불구하고 ext2 및 ext3 파일시스템이 복구될 수 있다.

단점

• JFS, ReiserFS, XFS 등에 비해 낮은 처리 속력

• 기능 (Functionality)

ext3는 ext2와 대부분 호환이 가능하도록 하는 것을 목표로 하였고, 많은 on-disk 구조들이 ext2의 on-disk와 비슷하다. 이 때문에, ext3는 inode의 동적 할당 및 다양한 블록 크기(frag와 tail)와 같은 최신 파일시스템 설계의 기능들이 부족하다. ext3 파일 시스템은 쓰기를 위해 마운트 되어있는 동안에는 fsck를 할 수 없다. 읽기-쓰기가 마운트 되어있는 동안 수집된 파일 시스템의 덤프 작업은 데이터 손상을 가져올 수 있다.

ext3는 JFS, ext4, 그리고 XFS와 같은 다른 파일 시스템에서 볼 수 있는 기능인 extents 기능을 지원하지 않는다.

• 조각 모음 (Defragmentation)

파일 시스템 레벨에서 사용할 수 있는 온라인 ext3 조각 모음 기능은 없다. e2defrag라고 하는 오프라인 ext2 조각 모음기가 있지만 ext3 파일 시스템은 ext2로 먼저 재변환되어야 한다. e2defrag는 데이터를 손상시킬 수 있다. 왜냐하면 e2defrag는 ext3의 새로운 기능들을 어떻게 다루어야 하는지 잘 알지 못하기 때문이다.^[3]

사용자 공간에서 이용할 수 있는 defragmentation 도구에는 Shake^[4]와 defrag^[5] 등이 있다. Shake는 전체 파일을 위한 공간을 바로 할당하며 단편화가 많이 되지 않도록 새롭게 파일을 할당하는 역할을 한다. 또한, 다음에 같이 사용되는 파일을 서로 쓸 수 있도록 한다. Defrag는 각 파일 스스로가 복사할 수 있도록 한다. 하지만 이러한 도구들은 파일 시스템이 비어 있을 때만 작동한다. 실제 조각 모음 도구는 ext3를 위해 존재하는 것이 아니다. ^[6] 'Linux System Administrator Guide' 에서는 "현재의 리눅스 파일 시스템은 연속적인 섹터에 저장될 수 없음에도 불구하고 서로가 파일 상에서 근접하게 모든 블록을 최소한으로 유지함으로써 단편화를 허용한다. 따라서 리눅스 시스템에서 단편화를 걱정할 필요는 없다." 라고 기술되어 있다.^[7]

전술한 것과는 상관 없이, 파일 단편화는 멀티미디어 서버 응용 프로그램에서와 같은 서버 환경에서는 매우 중요한 문제가 될 수 있다. ext3는 FAT 파일 시스템보다는 파일 단편화에 강한 편이지만 그럼에도 불구하고 ext3 파일 시스템은 시간이 지날수록 단편화가 더욱 진행된다. 결과적으로 ext3의 다음 버전인 ext4의 경우 파일 시스템 조각 모음 유틸리티를 포함하며 extents 또한 지원하게 된다. 속도가 빠르고, 동시적이며 랜덤한 파일 생성, 업데이트 및 접근이 일어나는 곳에서의 서버 응용 프로그램들은, (ext3와 같은) 일부 리눅스 파일 시스템에 조각 모음 기능이 없어서 큰 문제가 되기도 한다. 이러한 시스템에는 큰 규모의 carrier grade 음성 메일 시스템을 포함, Media-Messaging Service Centers(MMSCs) 및 SMS/SMSC(Short Message Service Centers) 서버도 포함된다. 규모가 큰 음성 메일과 같은 미디어 서버나 UMS 서버는 거의 실시간 상태로 수많은 사용자에게 음성 및 영상 스트림을 연결해주어야 한다. 이러한 타입의 응용 프로그램들은 파일 단편화가 이루어질 가능성이 있다. 음성이나 영상 파일을 재생하는 동안 미디어 파일 내에 많은 단편화 현상 때문에 접근 지연으로 재생 불능이나 재생 방해가 발생할 수 있다. 단편화 현상이 증가함에 따라, CPU 및 I/O 오버헤드 증가로 디스크 thrashing을 일으켰던 단편화를 가져오게 됨으로써 이러한 시스템들의 서비스 능력이 떨어지게 된다.

• 압축 (Compression)

ext3의 비공식 패치에서는 투명 압축이 지원된다. 이 패치는 e2compr의 직접적인 포트이며 개발이 더 필요한 상태이며, 업스트림 커널과 컴파일 및 부팅이 잘 되지만 저널링은 아직 구현되지 않았다. 현재 패치는 e3compr이며 다음 링크에서 확인할 수 있다: http://sourceforge.net/projects/e3compr/

• 크기 제한 (Size limits)

ext3는 개별 파일 및 전체 파일 시스템 상의 최대 크기에 제한을 두고 있다. 이러한 제한은 파일 시스템의 블록 사이즈에 따라 결정된다.^[8] (다음 차트 참조)

참고 8KiB 블록 사이즈는 8KiB 페이지(Alpha와 같은)를 허용하는 아키텍처에서만 가능하다. [영문 ext3 wiki] 여기와 내용이 틀림.

• Checksum을 검사하지 않는다. (No checksumming in journal)

Ext3는 저널에 기록할 때 checksum 검사를 하지 않는다. ‘barrier=1’이 마운트 옵션 (/etc/fstab)으로써 활성화되지 않고, 하드웨어가 캐시에 기록이 되지 않을 때, 충돌이 일어나는 동안 심각한 파일 시스템 손상의 위험을 일으킨다.^[9][10] (이 옵션은 대부분 모든 유명한 리눅스 배포판에는 기본적으로 비활성화 상태로 되어 있는데 이것은 대부분의 리눅스 배포판들이 이러한 위험에 노출되어 있다는 것을 의미한다.) 다음과 같은 시나리오를 생각해 볼 수 있다. 하드 디스크 쓰기가 제대로 작동하지 않는다면 (쓰기 속도를 향상시키기 위한 하드 디스크 캐싱 때문에), 하드 디스크는 다른 관련된 블록에 쓰기가 실행되기 전에 하나의 트랜잭션의 commit 블록을 종종 쓰게 된다. 다른 블록들에 쓰기가 되기 전에 전원이 잘못되거나 커널 패닉이 발생하면, 시스템은 재부팅을 해야만 하는 상태가 된다. 리부팅 시, 파일 시스템은 정상적으로 로그를 읽어 들여와서, winners (유효한 commit 블록과 함께 표시되도록 했던 유효하지 않은 트랜잭션을 포함하여 commit 블록이 있는 트랜잭션)를 재실행한다. 종료되지 않은 디스크 쓰기는 결과적으로 진행될 것이지만 손상된 저널 데이터를 사용하게 된다. 파일 시스템은 저널을 재실행하는 동안 손상된 데이터와 함께 정상적인 데이터의 중복 쓰기를 실행한다. 만일 checksum이 사용되었더라면 (상호 checksum으로 fake winner 트랜잭션의 블록이 표시가 된다면), 파일 시스템은 보다 더 잘 알게 되고 디스크 상에서 손상된 데이터를 다시 실행할 필요가 없다.

명세

같이 보기

• ext
• ext2
• ext4

주석

1. ↑ curious onloooker: Speeding up ext3 filesystems
2. ↑ Common threads: Advanced filesystem implementor's guide, Part 8
3. ↑ Andreas Dilger. Post to the ext3-users mailing list. ext3-users mailing list post.
4. ↑ Vleu.net: Shake
5. ↑ Index of /apps/defrag
6. ↑ RE: searching for ext3 defrag/file move program
7. ↑ http://www.tldp.org/LDP/sag/html/filesystems.html
8. ↑ Matthew Wilcox. Documentation/filesystems/ext2.txt. Linux kernel source documentation.
9. ↑ Re: Frequent metadata corruption with ext3 + hard power-off
10. ↑ Re: Frequent metadata corruption with ext3 + hard power-off
11. ↑ inode의 최대 개수는 파일 시스템 생성시 결정된다 (즉, 파일 및 디렉터리의 최대 개수). V는 바이트 단위의 볼륨 사이즈를 말하며, inode의 디폴트 개수는 V/213이고 (또는 블록의 개수, 어느 것이든 해당 개수보다 작다), V/223가 최소값이다. 디폴트 값은 대부분의 응용 프로그램에서 충분한 값이다. 한 디렉터리가 포함하는 하위 디렉터리의 최대 개수는 32000으로 고정된다.

바깥 고리

• Linux ext3 FAQ
• Introducing ext3 - IBM developerWorks Advanced filesystem implementor's guide, Part 7
• Ext2 File System For Windows GPL ext2/ext3 file system driver for Windows NT/2000/XP/Vista (opensource, supports read & write)
• Ext2 Installable File System For Windows ext2/ext3 file system driver for MS Windows NT/2000/XP (freeware, supports read & write on Windows NT4.0/2000/XP/2003/Vista on x86/AMD64)
• EXT2 IFS ext2/ext3 file system driver for MS Windows NT/2000/XP (opensource, doesn't support writing, doesn't support Windows XP SP2 or Windows Vista)
• Explore2fs An explorer-like GUI tool for accessing ext2/ext3 filesystems under MS Windows
• ext2/ext3 resizing tools
• Presentation on EXT3 Journaling Filesystem by Dr. Stephen Tweedie at the Ottawa Linux Symposium, 20 July, 2000
• Tutorial - Determining Your EXT3 Size Limits

ext

Developer	Rémy Card
Full name	Second extended file system
Introduced	January 1993 (Linux)
Partition identifier	Apple_UNIX_SVR2 (Apple Partition Map) 0x83 (Master Boot Record) EBD0A0A2-B9E5-4433-87C0-68B6B72699C7 (GPT)
Structures
File allocation	bitmap (free space), table (metadata)
Bad blocks	Table
Limits
Max file size	16 GB - 2 TB
Max number of files	1018
Max filename length	255 bytes
Max volume size	2-32 TB
Allowed characters in filenames	Any byte except NUL and '/'
Features
Dates recorded	modification (mtime), attribute modification (ctime), access (atime)
Date range	December 14, 1901 - January 18, 2038
Date resolution	1s
File system permissions	POSIX
Transparent compression	No (Available through patches)
Transparent encryption	No
Supported operating systems	Linux, BSD, Windows (through an IFS), Mac OS X (through an IFS)

The ext2 or second extended filesystem is a file system for the Linux kernel. It was initially designed by Rémy Card as a replacement for the extended file system (ext).

The canonical implementation of ext2 is the ext2fs filesystem driver in the Linux kernel. Other implementations (of varying quality and completeness) exist in GNU Hurd, MINIX 3, Mac OS X (third-party), Darwin (same third-party as Mac OS X but untested), some BSD kernels, in Atari MiNT, and as third-party Microsoft Windows drivers.

ext2 was the default filesystem in several Linux distributions, including Debian and Red Hat Linux, until supplanted more recently by ext3, which is almost completely compatible with ext2 and is a journaling file system. ext2 is still the filesystem of choice for flash-based storage media (such as SD cards, and USB flash drives) since its lack of a journal minimizes the number of writes and flash devices have only a limited number of write cycles. Recent kernels, however, support a journal-less mode of ext4, which would offer the same benefit along with a number of ext4-specific benefits.

Contents [hide] 1 History 2 ext2 data structures 2.1 Inodes 2.2 Directories 2.3 Allocating Data 3 File system limits 4 Compression extension 5 See also 6 References 7 Further reading 8 External links

History

The early development of the Linux kernel was made as a cross-development under the Minix operating system. Naturally, it was obvious that the Minix file system would be used as Linux's first file system. The Minix file system was mostly free of bugs, but used 16-bit offsets internally and thus only had a maximum size limit of 64 megabytes. There was also a filename length limit of 14 characters. Because of these limitations, work began on a replacement native file system for Linux.

To ease the addition of new file systems and provide a generic file API, VFS, a virtual file system layer was added to the Linux kernel. The extended file system (ext), was released in April 1992 as the first file system using the VFS API and was included in Linux version 0.96c. The ext file system solved the two major problems in the Minix file system (maximum partition size and filename length limitation to 14 characters), and allowed 2 gigabytes of data and filenames of up to 255 characters. But it still had problems: there was no support for separate access, inode modification and data modification timestamps.

As a solution for these problems, two new filesystems were developed in January 1993: xiafs and the second extended file system (ext2), which was an overhaul of the extended file system incorporating many ideas from the Berkeley Fast File System. ext2 was also designed with extensibility in mind, with space left in many of its on-disk data structures for use by future versions.

Since then, ext2 has been a testbed for many of the new extensions to the VFS API. Features such as POSIX ACLs and extended attributes were generally implemented first on ext2 because it was relatively simple to extend and its internals were well-understood.

On Linux kernels prior to 2.6.17,^[1] restrictions in the block driver mean that ext2 filesystems have a maximum file size of 2TB.

ext2 is still recommended over journaling file systems on bootable USB flash drives and other solid-state drives. ext2 performs fewer writes than ext3 since it does not need to write to the journal. As the major aging factor of a flash chip is the number of erase cycles, and as those happen frequently on writes, this increases the life span of the solid-state device.^[2] Another good practice for filesystems on flash devices is the use of the noatime mount option, for the same reason.

ext2 data structures

The space in ext2 is split up into blocks. These blocks are divided into block groups, analogous to cylinder groups in the Unix File System. There are typically thousands of blocks on a large file system. Data for any given file is typically contained within a single block group where possible. This is done to reduce external fragmentation and minimize the number of disk seeks when reading a large amount of consecutive data.

Each block group contains a copy of the superblock and block group descriptor table, and all block groups contain a block bitmap, an inode bitmap, an inode table and finally the actual data blocks.

The superblock contains important information that is crucial to the booting of the operating system, thus backup copies are made in multiple block groups in the file system. However, typically only the first copy of it, which is found at the first block of the file system, is used in the booting.

The group descriptor stores the location of the block bitmap, inode bitmap and the start of the inode table for every block group and these, in turn are stored in a group descriptor table.

Inodes

Every file or directory is represented by an inode. The inode includes data about the size, permission, ownership, and location on disk of the file or directory.

Example of ext2 inode structure:

Quote from the linux kernel documentation for ext2:

"There are pointers to the first 12 blocks which contain the file's data in the inode. There is a pointer to an indirect block (which contains pointers to the next set of blocks), a pointer to a doubly-indirect block (which contains pointers to indirect blocks) and a pointer to a trebly-indirect block (which contains pointers to doubly-indirect blocks)."

So, there is a structure in ext2 that has 15 pointers, the first 12 are for direct blocks. Pointer number 13 points to an indirect block, number 14 to a doubly-indirect block and number 15 to a trebly-indirect block.

Directories

Each directory is a list of directory entries. Each directory entry associates one file name with one inode number, and consists of the inode number, the length of the file name, and the actual text of the file name. To find a file, the directory is searched front-to-back for the associated filename. For reasonable directory sizes, this is fine. But for huge large directories this is inefficient, and ext3 offers a second way of storing directories that is more efficient than just a list of filenames.

The root directory is always stored in inode number two, so that the file system code can find it at mount time. Subdirectories are implemented by storing the name of the subdirectory in the name field, and the inode number of the subdirectory in the inode field. Hard links are implemented by storing the same inode number with more than one file name. Accessing the file by either name results in the same inode number, and therefore the same data.

The special directories "." and ".." are implemented by storing the names "." and ".." in the directory, and the inode number of the current and parent directories in the inode field. The only special treatment these two entries receive is that they are automatically created when any new directory is made, and they cannot be deleted.

Allocating Data

When a new file or directory is created, the EXT2 file system must decide where to store the data. If the disk is mostly empty, then data can be stored almost anywhere. However, performance is maximized if the data is clustered with other related data to minimize seek times.

The EXT2 file system attempts to allocate each new directory in the group containing its parent directory, on the theory that accesses to parent and children directories are likely to be closely related. The EXT2 file system also attempts to place files in the same group as their directory entries, because directory accesses often lead to file accesses. However, if the group is full, then the new file or new directory is placed in some other non-full group.

The data blocks needed to store directories and files can found by looking in the data allocation bitmap. Any needed space in the inode table can be found by looking in the inode allocation bitmap.

File system limits

Theoretical ext2 filesystem limits under Linux^[3]

Block size:	1 KB	2 KB	4 KB	8 KB
max. file size:	16 GB	256 GB	2 TB	2 TB
max. filesystem size:	4* TB	8 TB	16 TB	32 TB

The reason for some limits of the ext2-file system are the file format of the data and the operating system's kernel. Mostly these factors will be determined once when the file system is built. They depend on the block size and the ratio of the number of blocks and inodes. In Linux the block size is limited by the architecture page size.

There are also some userspace programs that can't handle files larger than 2 GB.

The maximum file size is limited to min( ((b/4)³+(b/4)²+b/4+12)*b, 2³²*b ) due to the i_block (an array of EXT2_N_BLOCKS) and i_blocks( 32-bits integer value ) representing the amount of b-bytes "blocks" in the file.

The limit of sublevel-directories is 31998 due to the link count limit. Directory indexing is not available in ext2, so there are performance issues for directories with a large number of files (10,000+). The theoretical limit on the number of files in a directory is 1.3 × 10²⁰, although this is not relevant for practical situations.

Note: In Linux kernel 2.4 and earlier block devices were limited to 2 TB, limiting the maximum size of a partition regardless of block size.

Compression extension

e2compr is a modification to the ext2 file system driver in the Linux kernel to support online compression and decompression of files on file system level without any support by user applications.

e2compr is a small patch against the ext2 file system that allows on-the-fly compression and decompression. It compresses only regular files; the administrative data (superblock, inodes, directory files etc.) are not compressed (mainly for safety reasons). Access to compressed blocks is provided for read and write operations. The compression algorithm and cluster size is specified on a per-file basis. Directories can also be marked for compression, in which case every newly created file in the directory will be automatically compressed with the same cluster size and the same algorithm that was specified for the directory.

e2compr is not a new file system. It is only a patch to the ext2 file system made to support the EXT2_COMPR_FL flag. It does not require you to make a new partition, and will continue to read or write existing ext2 file systems. One can consider it as simply a way for the read and write routines to access files that could have been created by a simple utility similar to gzip or compress. Compressed and uncompressed files coexist nicely on ext2 partitions.

The latest e2compr-branch is available for current releases of 2.6 and 2.4 Linux kernels, but development is stalled. There are also older branches for older 2.0 and 2.2 kernels, which are more stable.

See also

• e2fsprogs
• ext : ancestor of ext2
• ext3 : extended version of ext2
• ext4 : fourth extended file system
• StegFS : a steganographic file system based on ext2
• cloop
• SquashFS
• List of file systems
• Comparison of file systems
• Filesystem in Userspace (FUSE)

References

1. ^ linux/kernel/git/torvalds/linux-2.6.git/commitdiff:, [PATCH] 2TB files: add blkcnt_t, Author:Takashi Sato, 26 Mar 2006 09:37:52 +0000 (01:37 -0800) — Commit allowing for large files, git.kernel.org
2. ^ "Solid State Disk Drives". Super Talent Technology. http://www.supertalent.com/tools/ssd.php. 
3. ^ "File system guide". Linux Reviews. http://linuxreviews.org/sysadmin/filesystems/. 

Notes

• Card, Rémy. & Ts’o, Theodore. & Tweedie, Stephen (1994). "Design and implementation of the second extended filesystem". Proceedings of the First Dutch International Symposium on Linux.. ISBN 90-367-0385-9. http://web.mit.edu/tytso/www/linux/ext2intro.html. 
• Sourceforge e2compr project
• Sourceforge e2compr documentation
• Sourceforge e3compr project page, ext3 compression, alpha
• Dr. Dobb's Data Compression Newsletter Issue #46 - September 2003

Further reading

• John Newbigin. John's spec of the second extended filesystem. http://uranus.chrysocome.net/explore2fs/es2fs.htm. 
• Dave Poirier (2009). The Second Extended File System: Internal Layout. http://www.nongnu.org/ext2-doc/index.html. 
• Theodore Ts'o and Stephen Tweedie (June, 2002). "Planned Extensions to the Linux Ext2/Ext3 Filesystem". USENIX 2002 Annual Technical Conference. http://e2fsprogs.sourceforge.net/extensions-ext23/. 
• Larry Ayers (June 1997). "E2compr: Transparent File Compression for Linux". Linux Gazette (18). http://linuxgazette.net/issue18/e2compr.html. 
• Charles Cong and Jeremy H. Brown (1997-12-26) (PDF). A System for Transparent File Compression With Caching Under Linux. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.51.4780&rep=rep1&type=pdf. 
• Charles Cong and Jeremy H. Brown (1997-10-22) (PDF). A Survey of Modern File Compression Techniques. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.50.9847&rep=rep1&type=pdf. 

External links

• ext2fs user-space tools
• Read only software for ext2, ext3 on Windows
• Ext2read A windows application to read/copy ext2/ext3/ext4 files with extent and LVM2 support.
• A Free ext2 driver for Windows NT4.0/2000/XP/2003/Vista/2008 with full access to Linux Ext2 volumes (read access and write access). (Read-only access for ext3, too - see the FAQ)(Freeware)
• Ext2Fsd GPL ext2/ext3 file system driver for Windows 2000/XP/2003/VISTA/2008 (opensource, supports read & write, works with FreeOTFE)

exFAT

EFS Internals

EFS uses symmetric key encryption in combination with public key technology to protect files. File data is being encrypted with symmetric algorithm (DESX). The key, used in symmetric encryption is called File Encryption Key (FEK). The FEK in its own turn is encrypted with a public/private key algorithm (RSA) and stored along with the file. The reason why two different algorithms are used is the speed of encryption. The performance burden of asymmetric algorithms is too much to use them for encrypting a large amount of data. Symmetric algorithms are about 1000 times faster making their suitable for encrypting of large amounts of data.

As a first setp to encrypt file, NTFS creates a log file called Efs0.log in System Volume Information folder on the same drive, as encrypted file. Then EFS aquires access CryptoAPI context. It uses Microsoft Base Cryptographic Provider 1.0 as cryptographic provider. Having the crypto context open, EFS generate File Encryption Key (FEK).

The next step is to get public/private key pair; if it does not exist at this stage (the case when EFS invoked first time), EFS generate a new pair. EFS uses 1024-bit RSA algorithm to encrypt FEK.

Then, EFS creates Data Decryption Field (DDF) for the current user, where it places FEK and encrypts it with public key. If recovery agent is defined by system policy, EFS creates also Data Recovery Field (DRF) and places there FEK encrypted with public key of recover agent. A separate DRA is created for every recovery agent defined. Please note, that on Windows XP not included into domain, there's no recovery agent is defined, so this step is omitted.

Now a temporary file Efs0.tmp is created in the same folder as the file being encrypted. The contents of original file (plain text) is copied into temporary file, after that the original is overwritten with encrypted data. By default, EFS uses DESX algorithm with 128-bit key to encrypt file data, but Windows could be also configured to use stronger 3DES algorithm with 168-bit key. In that case FIPS compliant algorithms usage must be turned on in LSA policy (it is disabled by default):

EFS uses the registry to determine if it will use DESX or 3DES. If HKLMSYSTEMCurrentControlSetControlLSAFipsAlgorithmPolicy = 1, then 3DES will be used. If not, then EFS checks HKLMSoftwareMicrosoftWindows NTCurrentVersionEFSAlgorithmID (this value may not be present); if present, it will have ID CALG_3DES or CALG_DESX, otherwise, DESX should be used.

After encryption is done, temporary and log files are deleted.

After file is encrypted, only users who has correspondent DDF or DRF can access the file. This mechanism is separate from common security meaning that beside rights to access file, the file must have its FEK encrypted with user's public key. Only user who can decrypt FEK with his own private key, can access the file. The consequence is, that user, who has access to the file, can encrypt it thus preventing the owner to access his own file. Initially only one DDF is created for user who encrypts the file, but later he can add extra users to key ring. In this case EFS simply decrypts FEK with private key of user who wants to give access to the file to another user, and encrypts FEK with public key of target user, thus creating a new DDF which is stored along with the first one.

The decryption process is opposite to encryption:

First, system checks if user has a private key used by EFS. If yes, it reads EFS attributes and walk through the DDF ring looking for DDF for current user. If DDF is found, user's private key is used to decrypt FEK extracted from DDF. Using decrypted FEK, EFS decrypts file data. It should be noticed that file never decrypted in whole but rather by sectors when upper level module requests particular sector.

Recovery process is similar to decryption, except that it uses the recovery agent's private key to decrypt the FEK in the DRF, not in DDF:

DRA policy is implemented differently for Windows 2000 and Windows XP. In Windows 2000 by default on computers, not included into domain, local Administrator is added to Public Key Policy as Encrypted Data Recovery Agent. So, when user encrypts file, both DDF and DRF fields are created. If the last DRA is deleted, the whole EFS functionality is turned off and it is not possible to encrypt file anymore.

In Windows XP the situation is different. Since majority of home users working standalone do not need anybody else to be able to decrypt file except themselves, there's no need in data recovery agents, so there's no DRA included into Public Key Policy and EFS works without DRA. In this case only DDF field is created for encrypted file.

EFS - Encrypting File System. Encrypted Files and Folders
(NTFS ver. 3.0 and newer)

The Encrypting File System (EFS) provides the core file encryption technology used to store encrypted files on NTFS volumes. EFS keeps files safe from intruders who might gain unauthorized physical access to sensitive, stored data (for example, by stealing a portable computer or external disk drive).

Users work with encrypted files and folders just as they do with any other files and folders. Encryption is transparent to the user who encrypted the file; the system automatically decrypts the file or folder when the user accesses. When the file is saved, encryption is reapplied. Users who are not authorized to access the encrypted files or folders transparently receive an "Access denied" message if they try to open, copy, move, or rename the encrypted file or folder. The exact message text may vary depending on application which tries to access the file, because it is related not to user rights for file but to ability of EFS to decrypt file using user's private key.

EFS has the following benefits over 3rd party encrypting applications:

1. It is transparent for user and any applications. There's no risk for user to forget to encrypt file and leave data unprotected. Once file or folder is marked as encrypted, it will be encrypted in background without interaction with user. User does not need to remember password to decrypt files.
2. Strong key security. In contrast to other solutions when keys are based on user entered pass-phrase, EFS generates keys which are tolerant to dictionary based attacks.
3. All encrypting/decrypting processes are performed in kernel mode, excluding the risk of leaving key in paging file, from where it could be possibly extracted.
4. EFS provides data recovery mechanism which is valuable in business environment, giving an organization an opportunity to restore data even if the employee who encrypted it left the company.

Data Integrity and Recoverability with NTFS

NTFS is a recoverable file system that guarantees the consistency of the volume by using standard transaction logging and recovery techniques. In the event of a disk failure, NTFS restores consistency by running a recovery procedure that accesses information stored in a log file. The NTFS recovery procedure is exact, guaranteeing that the volume is restored to a consistent state. Transaction logging requires a very small amount of overhead.

NTFS ensures the integrity of all NTFS volumes by automatically performing HDD recovery operations the first time a program accesses an NTFS volume after the computer is restarted following a failure.

NTFS also uses a technique called cluster remapping to minimize the effects of a bad sector on an NTFS volume.

Important
If either the master boot record (MBR) or boot sector is corrupted, you might not be able to access data on the volume.

Recovering Data with NTFS

NTFS views each I/O operation that modifies a system file on the NTFS volume as a transaction, and manages each one as an integral unit. Once started, the transaction is either completed or, in the event of a disk failure, rolled back (such as when the NTFS volume is returned to the state it was in before the transaction was initiated).

To ensure that a transaction can be completed or rolled back, NTFS records the suboperations of a transaction in a log file before they are written to the disk. When a complete transaction is recorded in the log file, NTFS performs the suboperations of the transaction on the volume cache. After NTFS updates the cache, it commits the transaction by recording in the log file that the entire transaction is complete.

Once a transaction is committed, NTFS ensures that the entire transaction appears on the volume, even if the disk fails. During recovery operations, NTFS redoes each committed transaction found in the log file. Then NTFS locates the transactions in the log file that were not committed at the time of the system failure and undoes each transaction suboperation recorded in the log file. Incomplete modifications to the volume are prohibited.

NTFS uses the Log File service to log all redo and undo information for a transaction. NTFS uses the redo information to repeat the transaction. The undo information enables NTFS to undo transactions that are not complete or that have an error.

Important
NTFS uses transaction logging and recovery to guarantee that the volume structure is not corrupted. For this reason, all system files remain accessible after a system failure. However, user data can be lost because of a system failure or a bad sector.

Cluster Remapping

In the event of a bad-sector error, NTFS implements a recovery technique called cluster remapping. When Windows 2000 detects a bad-sector, NTFS dynamically remaps the cluster containing the bad sector and allocates a new cluster for the data. If the error occurred during a read, NTFS returns a read error to the calling program, and the data is lost. If the error occurs during a write, NTFS writes the data to the new cluster, and no data is lost.

NTFS puts the address of the cluster containing the bad sector in its bad cluster file so the bad sector is not reused.

Important
Cluster remapping is not a backup alternative. Once errors are detected, the disk should be monitored closely and replaced if the defect list grows. This type of error is displayed in the Event Log.

$EFS Attribute

When NTFS encrypts file, it sets flag Encrypted (0x4000) for the file and creates $EFS attribute for the file where it stores DDFs and DDRs. This attribute has Attribute ID = 0x100 in NTFS and can be pretty lengthy, occupying from 0.5K to several kilobytes depending on number of DDFs and DRFs.

Here's an example of $EFS attribute with more details.

$EFS attribute size

Computer SID and user number. It specifies folder where EFS stores certificates. In order to get folder name EFS makes some manipulations:

5A56B378 1C365429 A851FF09 D040000 - data stored in $EFS,

78B3565A 2954361C 09FF15A8 000004D0 - reversed

2025018970-693384732-167712168-1232 - converte to decimal

S-1-5-21-2025018970-693384732-167712168-1232 - SID prefix added

So, the folder will be %User Profile%Application DataMicrosoftCryptoRSAS-1-5-21-2025018970-693384732-167712168-1232

Public key thumbprint

Private key GUID (also used as container name). This name EFS uses when it aquires context from CryptoAPI provider. If there's only one DDFin $EFS attribute, container name can be figured out from $EFS (this field), but as more users added to the file (more DDFs or DRFs), PK GUID is not stored for all of them and must be retrieved from certificate storage based on public key thumbprint.

Cryptographic provider name = Microsoft Base Cryptographic Provider v.1.0

User name, to whom current DDF or DRF belongs

Encrypted FEK. Usually FEK is 128-bit long (in case of DESX) but since it's encrypted with 1024-bit RSA key, its encrypted length is 1024 bits.