Installing CentOS 5.1 in textmode

Need the old textmode installer? No time to kickstart?

Type
linux mem=192M noprobe

at the GRUB install prompt ;-). A system needs > 128MB RAM to install with Anaconda, but will default to the textmode installer if it has < 256MB RAM.

Skype VoIP on Solaris using BrandZ + CentOS Linux

Now that we've got a CentOS BrandZ Linux container and OSS audio drivers on Solaris, we can run Skype on the BrandZ container, and ssh -X skype. Of course, it's a lot easier if you're emulating Linux 2.6 and running CentOS 5 or some newer distribution, but let's stick to more "tested" software for the moment.



You're going to have to install the dependencies first:
# yum install libstdc++-ssa

And so on.

And you're going to have to use and older version of Skype. But no worries. At least it gets the job done.

Mounting Linux NFS shares in Solaris 10

First, export a filesystem on your Linux box (add it to the exports file, and exportfs). Make sure the kernel-nfs server service is started.

# /etc/init.d/nfs-kernel-server start

Edit your exports file:

# ed /etc/exports
i
/home/cmihai 192.168.1.13/24(rw,no_root_squash,subtree_check,async)
w
q

Export the filesystem:

# exportfs
/home/cmihai 192.168.1.13/24

# showmount -e
Export list for loonix:
/home/cmihai 192.168.1.13/24

On your Solaris 10 box, run:

# showmount -e loonix
export list for loonix:
/home/cmihai 192.168.1.13/24

It should give you results consistent with the results you got on your Linux machine.
Trying to mount the share as NFSv4 will fail (Linux NFSv4 isn't compatible with Solaris NFSv4).

# mount loonix:/home/cmihai /storage
nfs mount: mount: /storage: Not owner

So we're going to mount the share as NFSv3:

# mount -o vers=3 loonix:/home/cmihai /storage

# mount
/storage on loonix:/home/cmihai remote/read/write/setuid/devices/vers=3/xattr/dev=4840010 on Tue Mar 18 14:59:06 2008

BrandZ - Linux Branded Zones in your Solaris 10 Containers

You can easily create a Linux branded container and install Debian, CentOS or some other Linux distribution inside your Solaris container.

# zonecfg -z loonix
loonix: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:loonix> create -t SUNWlx
zonecfg:loonix> set zonepath=/export/loonix
zonecfg:loonix> add net
zonecfg:loonix:net> set address=192.168.21.73/24
zonecfg:loonix:net> set physical=bge0
zonecfg:loonix:net> end
zonecfg:loonix> commit
zonecfg:loonix> exit

We can install from a tar image, a CD or DVD or even a .iso file.

# zoneadm -z loonix install -d /export/home/cmihai/Desktop/centosimg/centos_fs_image.tar
Installing zone 'loonix' at root directory '/export/loonix'
from archive '/export/home/cmihai/Desktop/centosimg/centos_fs_image.tar'

This process may take several minutes.

Setting up the initial lx brand environment.
System configuration modifications complete.
Setting up the initial lx brand environment.
System configuration modifications complete.

Installation of zone 'loonix' completed successfully.

Details saved to log file:
"/export/loonix/root/var/log/loonix.install.4649.log"

# zoneadm list -iv
ID NAME STATUS PATH BRAND IP
0 global running / native shared
- loonix installed /export/loonix lx shared

# zoneadm -z loonix boot

# zoneadm list -iv
ID NAME STATUS PATH BRAND IP
0 global running / native shared
1 loonix running /export/loonix lx shared

# zlogin loonix
[Connected to zone 'loonix' pts/9]

Welcome to your shiny new Linux zone.

- The root password is 'root'. Please change it immediately.

- To enable networking goodness, see /etc/sysconfig/network.example.

- This message is in /etc/motd. Feel free to change it.

For anything more complicated, see:
http://opensolaris.org/os/community/brandz/

You have mail.
-bash-2.05b#

-bash-2.05b# passwd
Changing password for user root.
New UNIX password:
Retype new UNIX password:
passwd: all authentication tokens updated successfully.

-bash-2.05b# ed /etc/sysconfig/network
1d
1i
NETWORKING="yes"
HOSTNAME=loonix
.
w
q

Now we can ssh in :-).


Back in the global zone, a simple prstat -Z 1 1 reveals memory usage:

0 87 466M 568M 14% 0:40:04 5.7% global
1 11 7724K 12M 0.3% 0:00:00 0.0% loonix


As you can see, Solaris Zones are *very* lightweight.

There is also a Linux 2.6 BrandZ project, so we can easily run RHEL 5 or any other Linux distribution of our choice (with a bit of work).

Disk monitoring and tuning with dd and S.M.A.R.T. - Reallocating bad sectors and predicting disk failure

What is S.M.A.R.T.?

Modern disk drives will automagically reallocate bad sectors on the fly, as soon as they encounter some kind of R/W/ECC error. But in order for this to happen, it must first access that sector. This is why you never see surface errors on modern disks.

Modern hard drives (ATA and SATA) have S.M.A.R.T. - Self-Monitoring, Analysis, and Reporting Technology. Once you have that enabled in BIOS (assuming you have a S.M.A.R.T. capable disk and controller) you can monitor a number of disk health and performance parameters.

What you should keep an eye on is the Reallocated Sectors Count (if the drive has a problem with a R/W/ECC error it will mark the sector "Reallocated" and transfer the data to a spare area on the disk). This will result in some performance decrease, and is a sign of imminent disk failure.


Monitoring S.MA.R.T.

ATA and SATA disks:
To monitor S.M.A.R.T. data you can use HDTune on Windows or SmartMonTools (smartd, smartctl) on Darwin (Mac OSX), Linux, FreeBSD, NetBSD, OpenBSD, Solaris, OS/2, or eComStation systems. If you're up to it, you can also use SmartMonTools on Windows.

USB Enclosures:
While in most cases you should have no trouble using HDTune or SmartMonTools, some USB drive enclosures may be resilient to monitoring with S.M.A.R.T. programs and will require vendor software. In such cases, you can download vendor software to perform monitoring, like "Western Digital Data LifeGuard Diagnostics".

iPods:
You can also get S.M.A.R.T. info on your iPod. You can either configure it to act as a pass through device (regular USB media) or boot your iPod in diagnostic mode. You can check S.M.A.R.T. disk data and perform more test on your iPod. To do so, you must reset your iPod and hold REW + Select (5G) at the Apple boot menu. For other iPod models, see here (or Google Apple Diagnostic Mode your iPod Model).

Forcing the disk to remap damanged sectors

Now you should know that if you see any problems with Reallocated Sector Count, Reallocated Event Count, Seek Error Rate, Offline Uncorrectable, UDMA CRC Error Count, Multizone Error Rate, Hardware ECC Recovered values, you should consider getting a new disk. These are all signs of a failing disk. Learn more about S.M.A.R.T. attributes and their meaning here. Note that depending on vendor, there may also be enhanced or propriotary S.M.A.R.T. attributes. Read your HDD vendor documentation.

But sometimes you just need to get a bit more life out of a disk, and force the disk to reallocated damaged sectors. You can do so easily by performing a full raw disk read and write operation. For this, you can use the UNIX "dd" tool. Make sure your target disks aren't mounted (Type "mount" to list mounted disks then use "umount disk").

You can perform a disk read operation (reading the whole disk) using a syntax similar to:

# dd if=/dev/disk of=/dev/null bs=2048

You can perform a disk write operation (zero out the disk, this WILL result in data loss) using syntax similar to:

# dd if=/dev/zero of=/dev/disk bs=2048

Now you may wish to perform both a read and write at the same time, and not wipe out your disk data (zero it out). You can perform such a "disk refresh" using syntax similar to:

# dd if=/dev/disk of=/dev/disk bs=1m

This will read and rewrite the data to disk in 1MB chunks to prevent presently recoverable read errors from progressing into unrecoverable read errors.

Of course, you should read the dd manpage for your OS (on Windows you could use a dd for Windows implementation or resort to some sort of Linux or BSD LiveCD). Replace /dev/disk with your disk (make sure you're using the right disk). On Linux you can find out what disk you need to use from "dmesg" or /proc/partitions:

# cat /proc/partitions

You can also use "fdisk -l" to list partitons on your disk, see if that's the right disk

# fdisk -l /dev/hda

Do note that you need root permissions for all of this activity, so on some Linux systems you may need to use "sudo -i" to get a root shell, or precede all operations with "sudo".

While you're doing this rewrite operation, you should monitor the kernel log (dmesg). You can monitor /var/log/messages for this:

# tail -f /var/log/messages

You usually watch out for "DriveReady SeekComplete Error status=0x51 DriveStatusError error=0x04" or some other error.

You should also keep an eye on the Reallocated Sectors and other Interesting Parameters in smartctl:

# smartctl -A /dev/hda

Do this every now and then, and note the values before you've started the operation.

Once you begin the "dd" operations you can send dd a SIGINFO signal (use pkill / kill / whatever) to make it print out I/O information (progress). Some shells / TERMS also respond to Ctrl-T by sending SIGINFO.

# pkill -SIGINFO dd

Once you're done with dd and S.M.A.R.T. tools you should also perform a filesystem check (fsck / chkdsk / whatever).

Conclusions:

1. Monitor S.M.A.R.T. data with smartclt, keep an eye on Reallocs. Consider getting a new disk if you see reallocated sectors
2. Perform a disk refresh with dd in order to prevent recoverable read errors from progressing into unrecoverable errors. You don't need fancy tools like SpinRite.
3. You can use a simple Linux or BSD LiveCD to perform the disk refresh.
   
4. This is NOT a data recovery procedure. If you're doing data recovery, use something like dd_recover to a separate media.
5. This is NOT a step by step tutorial. Read your OS manpages to make sure you're not wiping out the wrong disk or something.
6. Always monitor S.M.A.R.T. parameters in order to spot disk failure before it happens.
7. Always keep backups.
   

Links and resources:

• Linux Harddisk Monitoring with SmartMonTools (smartctl)
• HDTune - S.M.A.R.T. tool for Windows
• Wikipedia S.M.A.R.T. page (good links)
  

WinDirStat - KDirStat graphical disk usage utility clone for Windows

If you're familiar with KDirStat (or even Baobab) Disk Usage Analyzers in *NIX, you'll be happy to know there's a clone for Windows (yes, WinDirStat is the clone). Makes disk cleanup so much easier.

Just grab WinDirStat from the SourceForce page, and run it. You'll get a nice graphical view of disk usage.


Anyway, if you use Linux/BSD/whatever just use Baobab (Gnome Disk Usage Analyzer - part of gnome-utils package), KDirStat part of KDE or Filelight.

You can also get Scanner if you like that Baobab / Filelight PIE chart view, get Scanner on Windows.

Free Veritas Cluster Server (VCS) Simulator

Veritas Cluster Server (VCS) is a High Availability (HA) cluster software for Linux, UNIX and Windows systems.

Symantec (they have bought Veritas) offers a freely available VCS emulator software that lets you train your skills :-).

You can also use the freely available documentation. Here is the Solaris Veritas Cluster Server (VCS) manual.

Of course, you could just grab the trial and run 2 Solaris (or Linux) virtual machines in VMware Server :-).

6000 CPU Linux Cluster in a single 20KW machine

"SiCortex 5832 is a 5-teraflop single-unit supercomputer."
Bet that got your attention.

"It uses low-power, custom 64-bit MIPS-processor packages, which are basically entire computers on a single chip. 5832 processor cores and 8TB of RAM in one chassis, which draws less than 20 kilowatts of power."



"The SiCortex systems are completely open source, even down to the microcode."

They even say it runs a modified version of *cough* Gentoo Linux and the (now Sun) Lustre Filesystem.

SiCortex also offers a 72 CPU desktop machine that's as big.. well, as a desktop machine :-).

Say hell yes to Desktop HPC:



Read more about their Kautz digraph based fabric and implementation here:
http://www.sicortex.com/products/white_papers

A brief look at ClamAV security - Open Source Antivirus

A brief (over)look at ClamAV security and performance. Comparing Open Source Antivirus products with commercial products.

I've been looking into signature based open source security products lately, namely Antivirus software (for SMTP E-Mail gateways or File Servers or anything that would be used as a distribution point for example as well as for Desktop systems).

I've had a pretty good look at ClamAV and other ClamAV based products (ClamWin or Spyware Terminator which include the ClamAV Engine) and found them rather weak, both from a security point of view (vulnerabilities) and in terms of detection rates, as well as performance (speed) and usability (interface, features, etc).

Clam AntiVirus (ClamAV) is an open source (GPL) anti-virus toolkit for UNIX (also ported to Windows, and used by GUI products such as ClamWin) designed especially for e-mail scanning on mail gateways. It provides a number of utilities including a flexible and scalable multi-threaded daemon, a command line scanner and advanced tool for automatic database updates. The core of the package is an anti-virus engine available in a form of shared library.


I. Security track record - a look at common vulnerabilities

So let's take a look at ClamAV's security track record. A simple look on Secunia at the vulnerability summary reveals a stunning 25 security advisories (1 unpattched), 31 % of which lead to System Access and 91 % are exploitable from remote. 40 % of vulnerabilities are "Highly Critical". I realize that some of these are in 3rd party plugins and compression tools and such, but when an attacker just sends a specially crafted archive via E-mail or whatever means, and manages to buffer overflow and gain system access when ClamAV scans it, that's when you need to look at other products. There are ways to mitigate it, with permissions, limited users, chroots, jails and such, but still...




So let's compare that with another security product, Avira Antivir, basically multi-platform Antivirus software (also has a free version for non-commercial usage). We can see only 2 security vulnerabilities reported, both local and privilege escalation (Windows only too).



Vulnerabilities range from Denial of Service: "A NULL-pointer dereference error exists within the "cli_scanrtf()" function in libclamav/rtf.c. This can potentially be exploited to crash ClamAV via a specially crafted RTF file." to Buffer Overflow and System Access: "An integer overflow error in rebuildpe.c when rebuilding PE files after unpacking can be exploited to cause a heap-based buffer overflow via a specially crafted executable. Successful exploitation of this vulnerability may allow execution of arbitrary code."


II. Vulnerability assessment tools - Static Code Analsys

What about a source code audit of ClamAV? Well, I don't really have the time for that, but I did parse it through Flawfinder, RATS and other Static Code Analysis tools looking for simple lexical "bad practices" and functions (strings functions for example) that may overflow buffers and so on.

So, I download the source code for the latest stable release: ClamAV 0.91.2 (signature) and stumble across a ton of bad programming practices. While most of the time, they mean nothing (as they aren't really vulnerabilities or even exploitable, they are usually where most errors occur, and, as such, should be avoided). We basically have hundreds of such occurrences (537 marked as High and 83 marked as Medium by RATS), so I'm just going to paste a few interesting examples here:

..\clamav-0.91.2/clamav-milter/clamav-milter.c:266: High: fixed size local buffer
Extra care should be taken to ensure that character arrays that are allocated on the stack are used safely. They are prime targets for buffer overflow attacks.

..\clamav-0.91.2/clamav-milter/clamav-milter.c:859: High: getopt_long
Truncate all input strings to a reasonable length before passing them to this function

..\clamav-0.91.2/shared/getopt.c:961: High: fprintf
Check to be sure that the non-constant format string passed as argument 2 to this function call does not come from an untrusted source that could have added formatting characters that the code is not prepared to handle.

..\clamav-0.91.2/sigtool/vba.c:1127: High: sprintf
Check to be sure that the format string passed as argument 2 to this function call does not come from an untrusted source that could have added formatting characters that the code is not prepared to handle. Additionally, the format string could contain `%s' without precision that could result in a buffer overflow.

..\clamav-0.91.2/clamav-milter/clamav-milter.c:1205: High: popen
Argument 1 to this function call should be checked to ensure that it does not come from an untrusted source without first verifying that it contains nothing dangerous.

..\clamav-0.91.2/clamav-milter/clamav-milter.c:1404: High: getenv
Environment variables are highly untrustable input. They may be of any length, and contain any data. Do not make any assumptions regarding content or length. If at all possible avoid using them, and if it is necessary, sanitize them and truncate them to a reasonable length.

..\clamav-0.91.2/shared/output.c:159: High: umask
umask() can easily be used to create files with unsafe priviledges. It should be set to restrictive values.

..\clamav-0.91.2/clamav-milter/clamav-milter.c:1585: High: gethostbyname
DNS results can easily be forged by an attacker (or arbitrarily set to large values, etc), and should not be trusted.

..\clamav-0.91.2/clamav-milter/clamav-milter.c:1703: High: strcpy
Check to be sure that argument 2 passed to this function call will not copy more data than can be handled, resulting in a buffer overflow.


..\clamav-0.91.2/shared/misc.c:132: High: printf
Check to be sure that the non-constant format string passed as argument 1 to this function call does not come from an untrusted source that could have added formatting characters that the code is not prepared to handle.

..\clamav-0.91.2/shared/output.c:235: High: syslog
Truncate all input strings to a reasonable length before passing them to this function

..\clamav-0.91.2/freshclam/manager.c:1307: High: system
Argument 1 to this function call should be checked to ensure that it does not come from an untrusted source without first verifying that it contains nothing dangerous.

..\clamav-0.91.2/sigtool/sigtool.c:815: High: strcat
Check to be sure that argument 2 passed to this function call will not copy more data than can be handled, resulting in a buffer overflow.

..\clamav-0.91.2/libclamav/hashtab.c:408: High: sscanf
Check to be sure that the format string passed as argument 2 to this function call does not come from an untrusted source that could have added formatting characters that the code is not prepared to handle. Additionally, the format string could contain `%s' without precision that could result in a buffer overflow.

..\clamav-0.91.2/shared/options.c:194: High: strncat
Consider using strlcat() instead.

..\clamav-0.91.2/shared/options.c:194: High: strncat
Check to be sure that argument 1 passed to this function call will not copy more data than can be handled, resulting in a buffer overflow.

..\clamav-0.91.2/shared/getopt.c:983: High: getopt
Truncate all input strings to a reasonable length before passing them to this function

..\clamav-0.91.2/shared/output.c:212: High: vfprintf
Check to be sure that the non-constant format string passed as argument 2 to this function call does not come from an untrusted source that could have added formatting characters that the code is not prepared to handle.

..\clamav-0.91.2/sigtool/sigtool.c:609: High: scanf
Check to be sure that the format string passed as argument 2 to this function call does not come from an untrusted source that could have added formatting characters that the code is not prepared to handle. Additionally, the format string could contain `%s' without precision that could result in a buffer overflow.

..\clamav-0.91.2/test/pe/debugpe.c:165: Medium: signal
When setting signal handlers, do not use the same function to handle multiple signals. There exists the possibility a race condition will result if 2 or more different signals are sent to the process at nearly the same time. Also, when writing signal handlers, it is best to do as little as possible in them. The best strategy is to use the signal handler to set a flag, that another part of the program tests and performs the appropriate action(s) when it is set.
See also: http://razor.bindview.com/publish/papers/signals.txt

..\clamav-0.91.2/libclamav/mbox.c:4659: Medium: getc
Check buffer boundaries if calling this function in a loop and make sure you are not in danger of writing past the allocated space.

..\clamav-0.91.2/sigtool/sigtool.c:172: Medium: read
Check buffer boundaries if calling this function in a loop and make sure you are not in danger of writing past the allocated space.

..\clamav-0.91.2/clamav-milter/clamav-milter.c:4260: Medium: stat
A potential TOCTOU (Time Of Check, Time Of Use) vulnerability exists. This is the first line where a check has occured. The following line(s) contain uses that may match up with this check: 4269 (open)

..\clamav-0.91.2/sigtool/vba.c:1063: Medium: realloc
Don't use on memory intended to be secure, because the old structure will not be zeroed out.

..\clamav-0.91.2/libclamav/lockdb.c:246: Medium: SetSecurityDescriptorDacl
If the third argument, pDacl, is NULL there is no protection from attack. As an example, an attacker could set a Deny All to Everyone ACE on such an object.

..\clamav-0.91.2/libclamav/msexpand.c:130: Medium: fgetc
Check buffer boundaries if calling this function in a loop and make sure you are not in danger of writing past the allocated space.

..\clamav-0.91.2/libclamav/others.c:433: Medium: srand
Standard random number generators should not be used to generate randomness used for security reasons. For security sensitive randomness a crytographic randomness generator that provides sufficient entropy should be used.

..\clamav-0.91.2/libclamav/others.c:697: Medium: lstat
A potential TOCTOU (Time Of Check, Time Of Use) vulnerability exists. This is
the first line where a check has occured.
The following line(s) contain uses that may match up with this check:
699 (rmdir), 715 (unlink)

..\clamav-0.91.2/contrib/Windows/Projects/clamAV/libclamav/regex.c:70: Medium: realloc
Don't use on memory intended to be secure, because the old structure will not be zeroed out.


You should keep in mind that ClamAV also relies on 3rd party libraries and tools, and it's security also depends on those. Again, I remind you that these aren't actual vulnerabilities, just bad practices that MAY lead to such vulnerabilities. You would need to look at the code and employ various testing tools to find them.

The authors of ClamAV should really solve these problems like OpenBSD developers do, even if it is something as simple as replacing strncat() with strlcat() - functions designed to be safer, more consistent, and less error-prone (this would be an issue on other platforms though, so it's not as simple as that).


III. Detection rates
So, here are some tests made by various research projects:

AV-Test is an anti-virus research project at the Institute of Technical and Business Information Systems at the Otto-von-Guericke University Magdeburg (Germany).

They measured the detection times for six of the malware programs released last week utilizing the MS05-039 Plug and Play vulnerability under 36 different anti-virus products. Eleven of the products were able to detect one or more of the attacks proactively, without any special pattern update to identify it specifically. Here are the numbers for each of the eleven:

Product Score
BitDefender 6 of 6 Fortinet 6 of 6 Nod32 5 of 6 eSafe 3 of 6 F-Prot 3 of 6 Panda 3 of 6 QuickHeal 3 of 6 McAfee 2 of 6 Norman 2 of 6 AntiVir 1 of 6 ClamAV 1 of 6

AV Comparatives did a test on various other Antivirus products not in their current testing process, including the ClamAV based ClamWin. Here are the results:
http://www.av-comparatives.org/seiten/ergebnisse/2ndgrouptest.pdf

This test also places ClamAV among the last in terms of detection rates:
http://www.sunbelt-software.com/ihs/alex/malwarereportjun3007.pdf

ClamAV also scores a 17 % (sig: 99% / heur: 1%) in retrospective Antivirus Performance Statistics, which also placed it among the last.

At VirusPool Tested Products ClamAV scores:

Number of descriptions in the database: 31928 out of 45159 live samples ( 70.7 %)
Number of 'in the wild' descriptions in the database: 25 out of 30 live samples ( 83.3 %)
(not very good, but not all that bad)

IV. Signature Updates:

How often is the clamav-virusdb updates? According to the FAQ, multiple times a week, and the response should be rather prompt. Anyway, to get an idea on that, take a look at http://lurker.clamav.net/list/clamav-virusdb.html

Conclusion:

I definitely need to investigate this further, but so far I find ClamAV to be highly overrated, simply based on the fact it is "part of the open source movement". It lacks a real time scanner (that's fair seeing how it was designed for mail gateways though), it has a horrible security track record, poor detection rates and dreadful performance. I somewhat doubt that it would last long as a commercial product. Still, it is available on multiple platforms, and the cost is just about right :-).

Things aren't all bad though (and I may have been to rash and only brushed the surface here). ClamAV is a free product (open source even), and it does leave a lot of room an opportunity to evolve. I just don't see this happening without powerful commercial backing. If you plan on using it though, make sure you've got at least another product scanning your emails :-).

Other links:
http://www.av-comparatives.org/
http://www.virustotal.com/

Mondo Backup - GPL Bare Metal Recovery Solution

Here's an interesting alternative to using CloneZilla and the likes for Bare Metal Recovery:

Mondo Rescue is a free (GPL) powerful disaster recovery suite for Linux (i386, x86_64, ia64) and FreeBSD (i386). It's packaged for multiple distributions (RedHat, RHEL, SuSE, SLES, Mandriva, Debian, Gentoo). It's basically the Linux equivalent of the powerful AIX mkcd / mkdvd.

It supports backups to tapes, disks, network and CD/DVD. It also supports multiple filesystems (ext2, ext3, JFS, XFS, ResierFS, VFAT and even NTFS), LVM, software and hardware Raid.

Example of using Mondo Rescue:

Generate a bootable DVD that also backs up /etc and can recover files running mondorestore:

# mondoarchive -OVr -d /dev/dvd -9 -I /etc -gF

Another interesting tool worth checking out is System Imager (automates Linux installs).

Recovering data from the EXT2 Filesystem

EXT2 Undelete: A pretty good tutorial for recovering data from ext2 filesystems:

http://fedora.linuxsir.org/doc/ext2undelete/Ext2fs-Undeletion.html

See also:

• Filesystem tools: Data Recovery and Data Carving (you can use Foremost, Scalpel or FTimes to carve out a LOT of data from a drive).
• The Second Extended File System - Internal Layout
  

Debian Etch AMD64 NVIDIA install one-liner

Add contrib and non-free repositories to sources.list:

cmihai@db:~$ cat /etc/apt/sources.list

deb http://ftp.lug.ro/debian/ etch main contrib non-free
deb-src http://ftp.lug.ro/debian/ etch main contrib non-free

deb http://ftp.lug.ro/debian-security etch/updates main contrib non-free
deb-src http://ftp.lug.ro/debian-security etch/updates main contrib n

Install the packages and configure X (nvidia-xconfig).

$ sudo aptitude install nvidia-kernel-2.6-`uname -r | sed 's,.*-,,g'` nvidia-settings nvidia-glx nvidia-xconfig nvidia-glx-ia32 && nvidia-xconfig

A few more tools to unerase files

A few free tools to restore deleted files:

• Undelete Plus works under Win 95/98/Me/NT/2000/XP/2003/Vista operating systems. The program supports all Windows file systems for hard and floppy drives including FAT12/16/32,NTFS/NTFS5 and image recovery from CompactFlash, SmartMedia, MultiMedia and Secure Digital cards.
• Recuva (pronounced "recover") is a freeware Windows utility to restore files that have been accidentally deleted from your computer.
• Restoration - free undelete tool for Windows.
• NTFS Undelete - free undelete tool for Windows.
• R-LINUX is a free file recovery utility for the Ext2FS partitions used in the Linux OS and several Unix. Host OS: Win9x/ME/NT/2000/XP/2003. Recovered data can be written to any disk visible by the host OS. R-Linux also can create DISK IMAGES that can be later processed by more powerful R-Studio. more about Data Recovery for Linux
• Ultimate Data Recovery - data recovery tool.
  
  

Paint.NET - tiny free Photoshop alternative for Windows or Mono

Paint.NET is free image and photo editing software for computers that run Windows. It features an intuitive and innovative user interface with support for layers, unlimited undo, special effects, and a wide variety of useful and powerful tools. An active and growing online community provides friendly help, tutorials, and plugins.



The programming language used to create Paint.NET is C#, with a small amount of C++ for installation and shell-integration related functionality. The source code is available under the terms of the MIT License.

The system requirements are:

• .NET Framework 2.0
• 500 MHz processor (Recommended: 800 MHz or faster)
• 256 MB of RAM (Recommended: 512 MB or more)
• 1024 x 768 screen resolution

The program itself is packed into a stunning 1.5MB, and is surprisingly fast and easy to use.

The program also runs on Linux and such (Solaris, MacOS, etc.) via Mono and makes a great light weight MIT licensed alternative to GIMP.

UNIX Deployment Tools - JumpStart, IgniteUX, NIM, KickStart, AutoYaST, FAI

Bare metal recovery and mass deployment tools for UNIX and UNIX-like systems:

On Windows there's RIS, WDS or tools like Ghost, on UNIX platforms we have tools like JumpStart, IgniteUX, NIM, FAI, KickStart, etc. to help with massive deployment of operating systems.

UNIX:

• Sun Solaris - Custom JumpStart and Advanced Installations - The custom JumpStart installation method is a command–line interface that enables you to automatically install or upgrade several systems, based on profiles that you create. The profiles define specific software installation requirements. You can also incorporate shell scripts to include preinstallation and postinstallation tasks. You choose which profile and scripts to use for installation or upgrade. The custom JumpStart installation method installs or upgrades the system, based on the profile and scripts that you select. Also, you can use a sysidcfg file to specify configuration information so that the custom JumpStart installation is completely hands-off.
• Sun Solaris - JumpStart Enterprise Toolkit: provides a framework to simplify and extend the JumpStart functionality provided within the Solaris Operating System.
  
• Sun Solaris Flash Archives (flar) - can be used with JumpStart technology to automate and speed up deployment or disaster recovery.
  
• HP HP-UX Ignite-UX - is an administration toolset that allows: Simultaneous installation of HP-UX on multiple clients, The creation and use of custom installations, The remote recovery of clients, The creation of recovery media.
• IBM AIX mksysb/mkcd/mkdvd: The mksysb command creates a backup of the operating system (that is, the root volume group). You can use this backup to reinstall a system to its original state after it has been corrupted. If you create the backup on tape, the tape is bootable and includes the installation programs needed to install from the backup.
• IBM AIX NIM - Network Installation Management - is an excellent feature of the AIX operating system and is very important for teams or companies that have a need to install or upgrade many RS/6000 machines with the same images at the same time. NIM supports the use of mksysb images. Performing a NIM mksysb installation is faster than performing a NIM rte installation, and with mksysb, you can optionally include other installed software. You can use a mksysb image to install the nodes of a CSM cluster.

Linux:

• RedHat Linux Kickstart provides automation of Linux installation that uses a single kickstart file to install the system on multiple machines.
• SUSE Linux AutoYaST - Automatic Linux Installation and Configuration with YaST2. AutoYaST allows unattended and automated installation. With AutoYaST, administrators can create a consistent baseline configuration for new installations in large or expanding deployments. In addition to AutoYaST, other installation methods include PXE Boot, CD-ROM, NFS, CIFS/SMB, HTTP, FTP, and the Service Location Protocol (SLP), which allows autodetection of install servers. ALICE, SuSEs former auto-installation system was a system built around the auto-installation features that were available with YaST1. In order to be able to use existing ALICE configuration files and resources, a special option is provided in the configuration system will let you convert ALICE configuration files into a control file readable by AutoYaST.
• Debian GNU/Linux FAI - Fully Automatic Installation - is an automated installation tool to install or deploy Debian GNU/Linux and other distributions on a bunch of different hosts or a Cluster. FAI can also be used for configuration management of a running system.
  

BSD:

• Automatic OpenBSD Installation - Jumpstart-style procedure for installing OpenBSD servers
• FreeBSD "JumpStart" Guide - This article details the method used to allow machines to install FreeBSD using the Intel PXE method of booting a machine over a network. Use sysinstall install.cfg for scripting.
• BSD PXEBoot - while not unassisted, BSD systems can easily boot from PXE and install over the network.
  

Tools:

• Cfengine - an adaptive system configuration management engine - is an automated suite of programs for configuring and maintaining Unix-like computers. It has been used on computing arrays of between 1 and 20000 nodes.
  

VMware - using named pipes to redirect serial console output

Most kernel debuggers (and some headless operating systems) require the use of a serial console. When running inside a VMware virtual machine, this could complicate things.

A neat way to solve this problem (and get your self a home made "LOM") is to use named pipes (well, it's actually a Unix Domain Socket) to redirect serial output to a TCP port.

Just add a Serial Port to your Virtual Machine, set the Virtual Machine end as the client, and the other end to an application.



Now we'll need to make use of a Named Pipe TCP Proxy Utility.

Windows users will probably pick NPTP (it has an easy to use GUI) and UNIX users (well, Linux, since VMware supports that) will pick something like Socat (an extended netcat that supports pipes, etc).



Some Windows user may prefer a command line alternative (with source code available). VMwareGateway also supports running as a service. It listens on localhost on port 567. The name of the pipe is \\.\pipe\vmwaredebug.




So, what about those who run VMware on Linux?

Well, we'll create a serial port in VMware, same as above, and point it to a file.
Then, we'll use socat:

% socat -d -d -d /path/to/port tcp4-listen:6666

See the socat manpage and examples for more info.


What about people who can't run VMware (eg: not running Linux or Windows)?

Well, qemu supports serial port redirection! See the '-nographic' option.

X forwarding with ssh -X and XMing

Redirecting X over tcp / udp and using something like "XWin -query machine" is insecure. So are most VNC solutions. You can use X forwarding over ssh:

% ssh -X -C -c blowfish MachineHostname xterm

(Blowfish encryption tends to be a bit faster, and -C enforces compression, though it should be enabled by default. If you have issues with X forwarding, check your /etc/ssh/sshd_config for "X11Forwarding yes" and "ForwardX11 yes").

Sometimes you _have_ to run a graphical interface on a UNIX machine, and do so from your Windows box. Examples include the Oracle Installer and some other Java applications.

While you could install Cygwin to provide you with an XOrg server implementation, Cygwin tends to be quite big. A lighter and faster option would be to use XMing. It's small, fast and just works. Just fire up XWin, and it will listen on 0:0 by default.



Now you can use "ssh -X machineIP xterm" and it will start a remote XTerm right on your machine. Or if you use putty, make sure you redirect X:




Now run your favorite GUI application.

Digital Forensic Tools: Imaging, Virtualization, Cryptanalysis, Steganalysis, Data Recovery, Data Carving, Reverse Engineering

"Jrypbzr gb gur bgure fvqr."

Computer Forensics is a science and an art. And to perform it, you need tools to identify, acquisition, preserve and analyze data in a clean, safe, non-destructive manner. Lots of tools. Everything from data acquisition to virtualization and steganalysis.

A list of more or less free tools (mostly open source or freeware, but I have included some relevant commercial products) no digital forensics expert should be without:

Data acquisition, enumeration, imaging and forensics tools: Toolkits and utilities.

1. The Sleuth Kit and Autopsy Browser. Both are open source digital investigation tools (a.k.a digital forensic tools) that run on Unix systems (such as Linux, OS X, FreeBSD, OpenBSD, and Solaris). They can be used to analyze NTFS, FAT, Ext2, Ext3, UFS1, and UFS2 file systems and several volume system types. The Sleuth Kit (TSK) is a collection of command line tools based on The Coroner's Toolkit (TCT). Autopsy is a graphical interface to TSK.
2. The Coroner's Toolkit (TCT) is a forensics toolkit for analysis of UNIX break-ins. It runs on BSD (OpenBSD, FreeBSD, BSD/OS), Solaris/SunOS, Linux and HP-UX.
   
3. WinHex: Computer Forensics & Data Recovery Software, Hex Editor & Disk Editor.
4. dcfldd is an enhanced version of GNU dd with features useful for forensics and security. GNU ddrescue is a data recovery tool. It copies data from one file or block device trying to rescue data in case of read errors. It's a better alternative to using dd_rescue and dd_rhelp or SpinRite (you can just do a disk refresh with dd: "dd if=/dev/disk of=/dev/disk bs=2m" while the drive isn't mounted - no write operations going on - or something along those lines in order to prevent presently recoverable read errors from progressing into unrecoverable read errors).
5. Sysinternals tools contains programs like streams that help us find data hidden inside alternate streams or strings that grep readable strings from a file. It also has tools like process explorer, procmon, autoruns and rootkit revealer that allow you to dig deep into the Windows operating system to process, disk and data related information.
   
6. Microsoft Log Parser is a powerful, versatile tool that you can use to extract information from files of almost any format by using Structured Query Language to text-based data such as log files, XML files and CSV files, as well as key data sources on the Windows operating system such as the Event Log, the Registry, the file system, and Active Directory.
7. AccessData Forensic Products: FTK - Forensic Toolkit, Registry Viewier - more neat tools from AccessData. Commercial products.
8. Clonezilla is used to clone many computers simultaneously. It can perform a full disk image or just file backup. It's a backup tool, but it can also perform bit by bit disk imaging.
9. Sysinternals LiveKD is a Live version of Windows Debugger (WinDBG) that allows you to run the Kd and Windbg Microsoft kernel debuggers, which are part of the Debugging Tools for Windows package, locally on a live system. This means that you to easily take a memory dump of a running Windows system (.dump /f YOURUSBDISK:\fullmemorydump.dmp). On UNIX systems you can use dd to take a snapshot of the system memory ("dd if=/dev/kmem of=/path/to/memorydump").
10. Paraben's Device Seizure - Cell Phone and PDA Forensic software. Specialized software for portable device forensics.
11. PDD is a forensic analysis tool for Palm OS platform devices. It is an open source Windows-based tool for Palm OS memory imaging and forensic acquisition. The Palm OS Console Mode is used to acquire memory card information and to create a bit-for-bit image of the selected memory region. No data is modified on the target device and the data retrieval is not detectable by the user of the PDA.
12. CDInfo is an application that will display all ISO descriptors from all attached cd-rom drives (Label, System, Application, VolumeSet, Copyright, Creation Date, Directory Start, Directory Length, extentions, tracks, etc).
13. PMDump is a Windows tool that lets you dump the memory contents (both RAM and swap) of a process to a file without stopping the process.

Virtualization: Once the actual machine is cloned, it's usually put inside a virtual machine (features like snapshots and debugging help quite a bit with the digital forensics process). This is called physical to virtual (P2V) migration.

1. Live View is a Java-based graphical forensics tool that creates a VMware virtual machine out of a raw (dd-style) disk image or physical disk. This allows the forensic examiner to "boot up" the image or disk and gain an interactive, user-level perspective of the environment, all without modifying the underlying image or disk. Because all changes made to the disk are written to a separate file, the examiner can instantly revert all of his or her changes back to the original pristine state of the disk. The end result is that one need not create extra "throw away" copies of the disk or image to create the virtual machine.
2. VMware Converter Starter is a free p2v (physical to virtual) migration tool. VMware Converter quickly converts Microsoft Windows based physical machines and third party image formats to VMware virtual machines. It also converts virtual machines between VMware platforms. Note: for digital forensic images, you should use LiveView, and not the converter.
   
3. VMware Server allows for free virtualization. You can use it in combination with Live View to virtualize existing environments, and use the snapshots feature to revert back to a previous state of a virtual machine in an instant.
4. QEmu - a much more flexible virtualization program, albeit a bit slower than VMware. Supports emulating IA-32 (x86) PCs, AMD64 PCs, MIPS R4000, Sun's SPARC sun4m, Sun's SPARC sun4u, ARM development boards (Integrator/CP and Versatile/PB), SH4 SHIX board, PowerPC (PReP and Power Macintosh), and ETRAX CRIS architectures. Also, qemu-img can be a valuable tool for converting virtual machine images. Also allows for some really low level debugging features. A modified version of QEmu can even emulate PIX platforms (or Juniper JunOS systems like Olive).
   
5. VirtualBox is a GPL licensed x86 virtualization platform that runs on Windows, Linux and MacOS hosts, and supports various x86 client machines (Windows, Linux, BSD, Solaris). It's a noteworthy alternative to using VMware, as performance tends to be pretty good.
6. Microsoft VirtualServer / Virtual PC are free virtualization products from Microsoft. They support all major features (snapshots, mounting ISO images and such), and performance tends to be reasonable (to some extent, similar to that of VMware).
7. SIMH is a highly portable, multi-system simulator. It can emulate VAX and PDP-11 platforms. Just in case you need to perform forensics on older minicomputers.
8. Hercules is an open source (QPL licensed) emulator of IBM Mainframe computers (System/370, ESA/390 architectures and even the 64-bit zSeries). Hercules runs under Linux, Windows (98, NT, 2000, and XP), FreeBSD, and Mac OS X (10.3 and later). Hercules will run OS/360, DOS/360, DOS/VS, MVS, VM/370, TSS/370 - all IBM public domain operating system, as well as OS/390, z/OS, VSE/ESA, z/VSE, VM/ESA, and z/VM, and even Linux/390 and Linux (SuSE, RHEL, Debian, CentOS and Slackware) on zSeries.
9. Oracle VM is a server virtualization software based on Xen and Oracle Linux (itself based on RHEL sources) that fully supports both Oracle and non-Oracle applications. It is a free alternative to VMware Virtual Infrastructure (VMware ESX + VirtualCenter). It is certified to run the Linux operating system, Oracle Database, Fusion Middleware, and Application software, thus makes a very good platform for investigating Oracle databases.
10. The Palm OS Emulator is a program based on the Copilot app that emulates the hardware of the various models of Palm-powered handhelds, making it a valuable tool for writing, testing, and debugging applications as well as obtainiwinng evidence from the device.
11. Microsoft Windows CE 5.0 Device Emulator contains the emulator technologies featured in Windows CE 5.0. By using the Device Emulator, you can run emulated-based images created by Windows CE 5.0 without installing Platform Builder, its platform development tool.
    

Password recovery tools: You may often need to recover keys and passwords.

"This text has been encrypted twice... for double protection!"

1. Ophcrack is a very efficient Windows password cracker based on rainbow tables. It will crack huge tables of LM hashes in under 3 minutes. It also comes in the form of a LiveCD (though in digital forensics cases it's usually best to extract the SAM file containing the password hashes from the disk image and use that. Ophcrack can be a lot more effective if you have more complete rainbow tables.
2. LCP is a free alternative to the now dead L0phtcrack.
3. John the Ripper is a very versatile password cracking tool. It's supported on different architectures and operating systems (UNIX, Windows, OpenVMS, etc) and it's quite fast. Besides several crypt(3) password hash types most commonly found on various Unix flavors, supported out of the box are Kerberos AFS and Windows NT/2000/XP/2003 LM hashes, plus several more with contributed patches.
4. Medusa is a very fast parallel brute force login password cracker.
5. Elcomsoft Password Recovery suite: anything from office, archives, pdf files etc. to email clients. These are commercial products though.
6. CmosPwd decrypts password stored in CMOS used to access BIOS SETUP. Works on a lot of BIOSes (AMI, Award, Phoenix, IBM, etc). It can also be used to backup, restore or erase the BIOS.
7. AccessData Decryption Tools: PRTK - Password Recovery Toolkit, DNA - Distributed Network Attack, PORT - Portable Office Rainbow Tables are some of the best and fastest tools in the business.
8. Offline NT Password and Registry editor - a utility to (re)set the password of any user that has a valid (local) account on your NT system. You do not need to know the old password to set a new one. Features a registry editor. Supports 32 and 64 versions of Vista (and NTFSv5).
9. Elcomsoft Distributed Password Recovery is designed for distributed recovery of forgotten or lost passwords of different documents. Version 2.0 adds support for Windows SYSKEY startup passwords, passwords stored in Domain Cached Credentials, includes updated Adobe Acrobat module, and provides hardware acceleration (now up to 25 times faster!) for NTLM password recovery using GeForce 8 video cards.
10. Dialupass - Dialup Password Recovery - Recovers the passwords of dialup entries (VPN and Internet connections) on Windows systems. NirSoft provides a couple of free password recovery tools for various products such as Instant Messaging applications, cached passwords stored by Internet Browsers, E-mail clients and so on.

Here's a little cool trick for recovering cached passwords (asterisk passwords) stored in your Internet Browser (Firefox, Opera, Internet Explorer or anything with JavaScript).

Steganalysis and stenography: how to detect hidden data using stenography.

1. Stegdetect finds hidden information in JPEG images using such steganography schemes as F5, Invisible Secrets, JPHide, and JSteg. XSteg is a GUI.
2. Stego Suite is a powerful commercial Stenography detection toolkit, consisting of 3 major tools.
3. Stegkit is an Automated Steganalysis Tool.
4. Digital Invisible Ink Toolkit is an open-source cross-platform image steganography suite that includes both steganography and steganalysis implementations.
5. StegSpy will detect steganography and the program used to hide the message.
   
6. SteGUI is a StegHide GUI.
7. Digital Watermarking allows you to hide copywrite information and such in media (images and such) that's present even after encoding to another format (bmp->jpg), printing, copy/paste, etc. You can use ImageMagick of various Photoshop plugins to do this.
8. Stepic is a Python module and command line tool for hiding arbitrary data within images by slightly modifying the colors. These modifications are generally imperceptible to humans, but are machine detectable.
9. wbStego4 offers steganography in bitmaps, text files, HTML files and PDF files. It is has two very user-friendly interfaces and is ideal for securely transmitting data online or adding copyright information, especially with the copyright information manager.
   
10. NL Stego is a system for text generation and text-based steganography. It combines Markov Models of several orders to generate random text resembling a given training text (or text corpus). It can also embed secret messages into pseudo-random generated text.
11. Steghide is an Open Source (GPL) steganography program that is able to hide data in various kinds of image- and audio-files. The color- respectivly sample-frequencies are not changed thus making the embedding resistant against first-order statistical tests. Supports compression of embedded data, encryption of embedded data, embedding of a checksum to verify the integrity of the extraced data and has support for JPEG, BMP, WAV and AU files.
    
12. StegFS is an Open Source (GPL) Steganographic File System for Linux. Not only does it encrypt data, it also hides it such that it cannot be proved to be there.
    

Filesystem tools: Data Recovery.

1. Undelete Plus is a free deleted file recovery tool that works for all versions of Windows (95-Vista), FAT12/16/32, NTFS and NTFS5 filesystems and can perform recovery on various solid state devices.
2. Avira UnErase Personal - a freely available unerase product.
   
3. TestDisk is a free (GPL) data recovery software that can fix partition tables, recover deleted partitions and rebuild NTFS boot sectors. It can find lost partitions (anything from BSD disklabels to IBM JFS, it supports pretty much anything).
   
4. GNU Parted is a program for creating, destroying, resizing, checking, and copying partitions, and the file systems on them. This is useful for creating space for new operating systems, reorganising hard disk usage, copying data between hard disks, and disk imaging. It can also be used to attempt recovery of the partition table similar to TestDisk (rescue START END).
   
5. Stellar Phoenix has various UNIX and *NIX (SCO OpenServer, Unixware, Sun Solaris, *BSD, HP-UX, MacOS) data recovery tools as well as some Windows Data Recovery tools. They are, however, commercial products.
6. R-Studio is a data recovery software suite that can recover files from FAT(12-32), NTFS, NTFS 5, HFS/HFS+, FFS, UFS/UFS2 (*BSD, Solaris), Ext2/Ext3 (Linux) and so on. This is a commercial product.
7. DiskInternals NTFS Recovery is a fully automatic utility that recovers data from damaged or formatted disks. This is a commercial product.
8. SystemRescueCD is a Linux system on a bootable CD/DVD for repairing your system and your data after a crash. It also aims to provide an easy way to carry out admin tasks on your computer, such as creating and editing the partitions of the hard disk. It contains a lot of system utilities (parted, partimage, fstools) and basic ones (editors, midnight commander, network tools). The kernel of the system supports most important file systems (ext2/ext3, reiserfs, reiser4, xfs, jfs, vfat, ntfs, iso9660), and network ones (SMB/CIFS and NFS).
9. PhotoRec is file data recovery software designed to recover lost files including video, documents and archives from Hard Disks and CDRom and lost pictures (thus, its 'Photo Recovery' name) from digital camera memory. PhotoRec ignores the filesystem and goes after the underlying data, so it will still work even if your media's filesystem has been severely damaged or re-formatted.
10. Datarescue PhotoRescue Advanced is picture and photo data recovery solution made by the creators of IDA Pro. PhotoRescue will undelete, unerase and recover pictures and files lost on corrupted, erased or damaged compact flash (CF) cards, SD Cards, Memory Sticks, SmartMedia and XD cards even when other solutions fail. Once the data is recovered, it guarantees its integrity. It supports recovery of all file types and is optimized for JPG, TIFF, GIF and BMP, as well as most camera RAW formats: CR2, RAW, RAF, CRW, NEF, ORF, MRW, etc and many types of movie files. In some cases, we can even rebuild pictures that have suffered minor corruptions.
11. MiTeC Windows Registry Recovery - crashed machine registry configuration data recovery.
    

Cryptography tools: Once the data has been collected, disks and media has been imaged, it now needs to be encrypted, hashed and digitally signed in order to be properly stored.

1. Truecrypt is a powerful open source encryption software that works on Windows (2000, 2003, XP, Vista) and Linux. It can do on the fly encryption, it can encrypt whole part