In my ongoing quest to get rid of unnecessary use of WinWord, I moved a two-page checklist into our internal wiki. Unfortunately I came across two problems while doing that:

1. HTML <li>s don't support big empty circles. They default to black bullets, and the CSS {"list-style-type: disc;} only creates tiny circles.

   Unicode to the rescue: \x25EF, also known as \N{LARGE CIRCLE} or ◯ makes a fine replacement. Two lines of CSS to fix your list:

   ul { list-style-type:none; }
   ul li:before { content: "\25EF"; }
   

2. No page breaks. The list was on two pages for a reason and I needed to keep it that way. That was easily fixed by inserting

   <br class="break" /> 
   

   and making the break with CSS:

   br.break { page-break-before: always; }
   

   My first attempt was to use a <hr> instead – but strangely enough it made Firefox print another empty page at the beginning when used inside our wiki.
   

Putting it all together for TWiki:

<!-- <pre> -->
<style type="text/css" media="all">
br.pagebreak { page-break-before: always; }
.patternTopic > ul { list-style-type:none; }
.patternTopic > ul > li:before { content: "\25EF"; }
</style>
<!-- </pre> -->

The .patternTopic makes it only apply within the article, and the > makes it only apply on the first level after that.

– Sec

Last time we discussed the block order of linux-mdraid.

How do you find out which block order your RAID has?

The simplest way requires a working RAID to test against. (Ray created a small (50MB) test-RAID for that). First we get the first few blocks from each raw disk:

for $disk in sda1 sdb1 sdc1 sdd1 ;do
  for $nr in 0 1 2 3 4;do
    dd if=/dev/$disk skip=$nr count=1 bs=16k of=B.$disk.$nr
  done
done
dd if=/dev/md0 bs=16k count=20 of=RAID

Note that this assumes your stripe-size is 16k. If you know it is different, change it, if not you will find out later and have to retry with an adjusted value.

Now try to match up the first block with the RAID contents like this:

cat B.sda1.0 | cmp - RAID

If the block matches, you will get:

cmp: EOF on stdin

If the block is the wrong one:

stdin RAID differ: char 1, line 1

If none of your first blocks (the files ending with .0) match, either your block-size is too big (try again with half the previous size) or your RAID prefixes the disks with some internal bookkeeping info (in that case you can try to start with later blocks)

Now try to match the next blocks by adding then one by one to the cat command line like this:

cat B.sda1.0 B.sdb1.0 | cmp - RAID

that way you will easily recover the block allocation order of your RAID.

For example our linux-mdraid starts like this:

cat B.sda1.0 B.sdb1.0 B.sdc1.0 B.sdd1.1 B.sda1.1 ...

After that, its only a two-line patch to raidextract to fix that – hope you know C ;-)

That concludes our intermission for today. Tomorrow we will see why all this work wasn't even necessary.

– Sec

We left our heroes yesterday with a broken RAID 5 due to read errors on multiple disks. (Read part I: here)

A great starting point is raidextract by Peter Benie which attempts to re-assemble a RAID5. His web-page on this tool also serves as a great overview in the inner workings of standard RAID 5.

The first problem we stumbled upon is the fact that it assumes a certain pattern of the Parity blocks. (All examples from here on assume 4 disks, since that is what we had. But of course that's all applicable to any number of disks)

Adapting the example from his page:

Our Linux-mdraid unfortunately didn't conform to this expectation. Not only does it start with parity on the last disk (which raidextract would support with --rotate), but it also moves the parity block 'backward' instead of 'forward'.

The correct allocation order looks like this:

A quick&dirty hack to raidextract to implement this order:

--- raidextract.c	2008-07-26 11:33:53.000000000 +0200
+++ raidextract-new.c	2010-06-28 13:49:54.000000000 +0200
@@ -316,8 +316,10 @@
 	int paritydisk=(stripe / (disks-1) + rotate) % disks;
 	int len=stripesize-offset;
 	int bytes;
+	int ndisk;
 	char *ptr;
 
+	ndisk=(disk-paritydisk+3)%disks;
 	if (!noparity && paritydisk <= disk) disk++;
 	if (len>raidlen) len=raidlen;
 	if (winoffset+len > datasize) len=datasize-winoffset;
@@ -337,7 +339,7 @@
 	raidstart+=len;
 	raidlen-=len;
 
-	ptr=window[disk][windowalt]+winoffset;
+	ptr=window[ndisk][windowalt]+winoffset;
 	while (len)
 	{
 		bytes=write(STDOUT_FILENO, ptr, len);

A run on an error-free test-RAID confirms this and extracted it correctly. Yay!

continued in part III, coming soon

– Sec

The Munich CCC fileserver uses (as many other servers) software RAID 5 amongst its disks. We all (should) know that RAIDs are no substitute for backups, which was reinforced by a recent problem we had. While RAID level 5 can recover gracefully from a single failed disk, it generally can't cope with multiple failed disks at the same time.

One of the problems with large harddisks is, that there may be yet undetected errors on it, just because you haven't attempted to read that part for quite some time. Now when you start a rebuild of a RAID5, these errors quickly pop due to the rebuild process needing to read all the data. This is the main reason why you should regularly run complete surface scans on your RAID arrays.

Almost all RAID implementations tend to mark a whole disk as failed as soon as it contains a single error. This becomes a problem as soon as you detect a second error on your currently degraded RAID you are just attempting to rebuild.

Fortunately there is still hope. If the errors on your failing disks occur on non-overlapping points of the array, you can recover a complete copy of your data by assembling just the right pieces. But unfortunately there appears to be no hardware or software RAID solution able to do that out of the box. So we're left to try this manually.

more on this saga in part II, coming soon…

– Sec

Für alle dies bisher nicht mitbekommen haben, die Kurzzusammenfassung:

Auf der Easterhegg gabs als Badge das Haserl. Fertig bestückt mit einer kleinen analogen Schaltung die die LEDs bei Dunkelheit leuchten lässt.

Mehr Text und Bilder hier...
posted at: 22:20 | Category: /tidbits | permanent link to this entry | 0 comments (trackback)

Danke zu freundlich, aber ich will wirklich nicht

So gings mir heute mit Ubuntu. Ich wollte einen sshd installieren, aber auch wirklich nur einen sshd, und kein X11. Ubuntu ist aber wohl leider der Meinung ein X11 gehört auf jedes System. apt-get install openssh-server sagt u.a.:

Was tun? Nach ein bisschen frickeln bin ich auf folgende, an sich recht einfache, Lösung gekommen:

Voilà. Kein Plattenplatz im chroot für nix verschwendet. Yay!

– Sec

The Greasemonkey-script to insert xkcds image title tags broke yet again. This time Randall Munroe added a link around the image which my script didn't expect. Seeing that the last breakage is over a year in the past, I think this is a good track record :-)

Without much further ado, here is the fixed version: xkcd-titles.v4.
– I hope it works for you :)

– Sec

(Oder auch: Hilfe, mein Echofon ist nicht mehr rechts :-)

Ich bin ja seit längerem Fan von echofon (also, damals als es noch Twitterfox hieß). Beim letzten Update/Neustart von Firefox war das Icon von Echofon nicht mehr rechts unten in der Ecke der Statuszeile sondern deutlich weiter links. Verdrängt von lauter anderen Extension-Icons (NoScript, Operator, HTML Validator, Greasemonkey, Firebug, FoxyProxy, CookieSafe, Ghostery, RefControl, Adblock Plus, NoSquint, Read It Later)

(Wann gründet eigentlich mal jemand die Selbsthilfegruppe anonymer Extension-Süchtiger?)

Naja, zumindest war das Icon nicht mehr rechts, was ich ja noch verschmerzen könnte, aber auch das Popup-Window war über dem Icon, und somit Mitten in der Seite.

Nun, was tut der findige Geek? Nachdem man die Position nicht konfigurieren kann, muss sie wohl an der Startreihenfolge der Add-Ons liegen. Also flugs mal Echofon ans ende der Liste gepackt, und siehe da, beim nächsten restart ist Echofon wieder brav am rechten Rand. Yay!

Nun einfach Echofon mit dem letzten Platz der Liste vertauschen. Nicht vergessen die Nummer am Anfang der Zeile anzupassen, und fertig.

– Sec

Grade geisterte durch die Medien das eine Liste mit 10000 hotmail Passwörtern aufgetaucht sei. Das hat mich als Sysadmin natürlich gleich interessiert.

10 Minuten, und einen Bittorrent-Client später hatte ich dann auch ein File:

Nun aber die Preisfrage weshalb mich das File interessiert hatte:

Was ist das häufigste Passwort in der Liste?

Mehr nach dem Weiterklick...
posted at: 19:26 | Category: /tidbits | permanent link to this entry | 1 comment (trackback)

Unter Ubuntu gibt es mittlerweile eine sogenannte Bildschirmlupe die man hervorragend aus Versehen aktivieren kann. In Wirklichkeit ist das aber keine Lupe, denn es wird einfach nur der gesamte Bildschirm um ca. Faktor 4 vergrößert. Verstellen kann man die Vergrößerung dann per Windows+Mausrad.

Wer das loswerden will, dem sei hier ein:

und/oder ein

empfohlen.

– Sec

Just now I tried to update the (sadly neglected) FreeBSD installation on my Laptop. One of the bigger changes was the upgrade to Xorg 7.4 which introduces hald. Hald, for those who don't know, is the Hardware Abstraction Layer Daemon, a linuxy invention which spreads it tentacles throughout your system to fix problems FreeBSD didn't yet have – But I digress.

After starting the freshly updated X server you will be in for a treat. It starts all right, but nothing will work, neither keyboard nor mouse. Not even Ctrl+Alt+Backspace.

Ok. Ctrl+Alt+F1 works. So get back to the console and kill the X server.

The problem is, Xorg now tries to use hald by default and if it isn't running, you loose.

The quick fix is to add the following to your xorg.conf:

Section ServerFlags
Option AutoAddDevices off
EndSection

But if you want to do it the new way...
– Sec

Just now I noticed that with flashblock enabled, I can't view any videos on the sevenload.com web page.

Some debugging later I found out that they have a tag in their CSS which explicitly sets that very <div> that flashblock internally creates to visibility: hidden;.

I am unsure whether this is just some unlucky circumstance or an explicit attempt to annoy users of the flashblock extension. Nonetheless, this needs to be fixed.

So the obvious thing (after filing a quick bug report with the flashblock people) was to fix that problem. Greasemonkey to the rescue!

Have fun with the resulting small flashblock-fix userscript.

– Sec

P.S.: (15.3.09) the new unstable version of flashblock already contains a fix for that problem. Hooray for quick reaction time.

A few days ago knarf noticed something strange on one of his boxes:

netstat showed an open socket, but both sockstat or lsof didn't list it at all.

A quick test with telnet confirmed that the port was indeed open.

The port number did not ring a bell - although it was below 1024 (so it was a privileged port) - it was not listed in /etc/services.

Trying to find out where that socket came from we discussed a few ideas - starting with the obvious: A rootkit. It would be unlikely that both sockstat and lsof would be patched to skip that socket and netstat would've been forgotten, but it was still an option.

The second idea was, that lsof/sockstat didn't show anything, because there was no process there. – As the socket did react to connection attempts, this meant that the kernel had to be involved.

Checking the system include headers, the struct socket was quickly located and contained an interesting looking field named so_upcall.

To look at the socket in question, we first need to find it in memory. The easiest way is to get netstat -A. This command gives us the address of the tcp control block (tcpcb) for tcp sockets or the address of the internet protocol control block (inpcb) for udp sockets:

netstat -Aan | grep ' \*\.portno .* LISTEN'

The tcpcb structure contains a pointer to the inpcb structure which in turn has a pointer to the stuct socket which we're currently interested in.

So, lets fire up kgdb and look at the so_upcall element:

p ((struct tcpcb*)0xc1a6bde0)->t_inpcb->inp_socket->so_upcall (for tcp)
p ((struct inpcb*)0xc1a6bde0)->inp_socket->so_upcall (for udp)

A quick test on the ssh socket returned:

$1 = (void (*)(struct socket *, void *, int)) 0

Which is the expected result, as there is an actual process listening, and no upcall necessary.

Our mysterious socket returned:

$2 = (void (*)(struct socket *, void *, int)) 0xc0974250 <svc_vc_soupcall>

So now we can be sure that it is the kernel listening here.

Another quick search for the svc_vc_soupcall function returns a hit in rpc/svc_vc.c pointing us to the source of the socket: The sunrpc subsystem.

And 'lo and behold, a call of rpcinfo listed the strange socket as belonging to nlockmgr, the locking manager for nfs.

Case closed.

- Sec

Im Rahmen eines kleinen Seitenprojekts auf dem diesjährigen CCC Congress habe ich mit monoalphabetischen Subsitutionschiffren rumgespielt. Und weil das per hand immer so nervig ist, ist im Laufe des Abends erstmal ein kleines (Curses-basiertes) Perlscript entstanden. Neben den offensichtlichen Features Zeichen interaktiv zu ersetzen (und wieder zu löschen) und dem Anzeigen der Frequenzverteilung kann es auch intelligent in einem Wörterbuch suchen.

Gestern habe ich bei Bruce Schneier im Blog den Hinweis zu einem Rätsel auf der FBI-Webseite gefunden, und auch dieses war in wenigen Sekunden gelöst.

Falls noch andere Leute gerne mit so etwas spielen, veröffentliche ich hier den gegenwärtigen Stand des Perlscripts. Schönheitswettbewerbe wird es wohl nicht gewinnen, aber es erfüllt seinen Zweck, und perlcritic meckert auch nur 2 Dinge an ;)

 -rwxr-xr-x  1 sec  sec  7279 Jan  8 23:40 cwb.pl

Für Patches und Verbesserungen bin ich natürlich jederzeit zu haben, viel Zeit werde ich aber wohl nicht mehr investieren.

Achja, Qwereqq Tyei ipofy igwhj: Mpi Qnytm nteiif zyejyj Oxtzyjiprpf

– Sec

P.S.: Wenn das Perlscript nicht läuft, ist dein Fenster zu klein %-)

The Greasemonkey-script to insert xkcds image title tags broke again, and needed another update. Looks like Randall Munroe is actively trying to make my life harder than it already is :).

So here is an enhanced version: xkcd-titles.v3 - I hope it works for you :)

– Sec