Fefe bloggt in letzter Zeit lauter Links in die Arte Mediathek. Leider gibt es den Kram dort nur gestreamt, und nicht zum runteladen.

Dafür gibt es zwar das praktische rtmpdump, aber man muss immer noch die URL des Streams rausfummeln.

Zusätzlich dazu gibt es noch eine pseudo-Security in der der Client einen passenden Hash mitschicken muss, sonst bricht der Stream nach kurzer Zeit ab.

Weil mich das schon beim zweiten mal genervt hat, hab' ich ein kleines Script geschrieben, was sich für einen durch die .xmls wuselt, und die passende Kommandozeile für rtmpdump zusammenbastelt.

Die komfortable Version ist ein Greasemonkey-Script arte_rtmpdump.user.js das beim betreten der Mediathek-Seiten einfach in einem kleinen Window den passenden Aufruf von rtmpdump anzeigt.

Wer nicht so auf Browser-Integration steht kann mein kleines perl-script arte_rtmpdump.pl nehmen, das braucht allerdings noch XML::LibXML und mein GET.pm als Module.

Happy downloading…

– Sec

Update: I fixed arte_rtmpdump.pl to now only require the Mediathek URL on the commandline and work from there. So no more checking the source :-)

The following post is a bit longish, and details my foray into the BSD TCP/IP stack debugging and finding what I think is a 15-year old bug.

Read more ...
posted at: 14:43 | Category: /tidbits | permanent link to this entry | 6 comments (trackback)

At work, we acquired the above mentioned 24Port switch from Dell. It supports a so called managed mode (needs to be enabled by poking into a hole on the switch with a paperclip).

Enabling that sets the switch IP to 192.168.2.1/24 – with no way to change that. So far so annoying. Ok, I can configure an alias on my Ethernet to connect to that.

When you do that, don't be impatient, the IP needs ages to come up, but eventually you will be able to ping it.

I was a bit surprised that it actually only supports http on port :80, no https, no ssh, no telnet and no snmp. – But ok, lets log in via http.

The login page is not only a <frameset> (well, it came out 2005, but was that really necessary?), but only works with javascript: – WTF?

Trying to login with the default credentials (user admin, empty password) results in a Error message User name or Password is missing.

Gory details about this bug...

TL;DR: Their TCP/HTTP code is broken, and their support doesn't know it.

Have fun.

– Sec

Not very exciting news, but as a result of a recent discussion on Math symbols:

In case you need to use them anywhere, these are the codes:

More Math fun in the Unicode space can be found here

Practice safer math,
– Sec

I like mplayer as a very versatile video player, just the documentation is a bit lacking at times. This is my scratchpad of options I've found useful over the years:

-af scaletempo

To keep the pitch on your video if you play it faster/slower - Very useful to speed up slow lectures/talks and still understand what's being said.

Use [ or ] to increase/decrease video playback speed.

-af volume=20

Increase the volume a bit, useful to boost audio over the standard maximum.

-vm -adapter 2

Play fullscreen on your second video output.

-ao null

Play without audio.

If you know any other interesting options, please share them below..

– Sec

As most of you already know, I own an android mobile. Recently I wanted to add an RSS feed URL to my android podcast client, but was annoyed by the fact that I had to copy this long URL by hand. I would rather copy&paste it.

As I have rooted my phone, it had to be possible one way or another. Looking around, I found no usable solutions.

The most general solution would be to simulate keyboard input. Digging around a bit in Linux internals I found a way to inject key-presses via /dev/input/events* so I wrote a little program: btype.c. For those not wanting to compile it themselves, you can download the binary here.

It will accept text on the command-line, or read interactively from stdin.

Writing it was actually more difficult than I expected - I didn't want to hardcode the keyboard scancodes, so I had to read&parse the keymap and keylayout files which is always onerous in C. Also I learned funny ioctl%s to find the correct device in %/dev/input and to switch a tty to single-character input. Last but not least I had to find out how to get an android property value in C. – Hopefully all this made the program more portable to other android devices.

I hope someone finds a good use for it, let me know if you like it,
Sec

As regular readers here will know, I'm a big fan of the Greasemonkey extension - It allows the hacker in you to play with pages to your hearts content. Visit http://userscripts.org/ to see what people have created with it.

But today I wanted to tell you about Stylish. It presents a nice UI for adding custom URL-based stylesheets. Okay, granted, nothing you couldn't do with Greasemonkey if you wished, but it's really easy and quick to do.

Case in point today: The Android-Developers Blog has an annoying problem. If you visit it while using NoScript, it disables scrolling so you can't read the whole article without enabling JavaScript. – Stylish to the rescue:

@namespace url(http://www.w3.org/1999/xhtml);

@-moz-document domain("android-developers.blogspot.com") {
  html, body {
    overflow: visible !important;
  }
}

On a related note: If you always wanted to know what exactly !important does in CSS, check here

– Sec

Today, a colleague asked me to help him extract a logo as a file from an openoffice document. This is a task which seems easy enough, given that .odt documents are essentially zip files.

Extracting the .odt revealed (among other files) two interesting files: Object 1 and ObjectReplacements/Object 1. Using file to determine the file-types was quite unhelpful - on two different machines I got:

Object 1: Microsoft Office Document
Object 1: CDF V2 Document, corrupt: Cannot read summary info

And the other file stays enigmatic:

ObjectReplacements/Object 1: data

The ObjectReplacements file starts out with

0000000: 5643 4c4d 5446 0100 3100 0000 0000 0000  VCLMTF..1.......

which some googling reveals to be a StarView Meta file. - This is an openoffice internal format, supposed to have the extension .svm and can be opened by OO Draw.

But I wanted to get at the original file. Both suggestions from file(1) are wrong, but the Microsoft Office Document actually points in the right direction…

Checking in META-INF/manifest.xml gives us the supposed mime-type of application/vnd.sun.star.oleobject and further googling shows us that this is an so called OLE Compound File.

Now while I could easily find a Windows program to parse this file, I found no such thing for Unix. – Which lead me to a quick hack using perl and OLE::Storage_Lite to crate cfx the compound file extractor.

ice:~/ole>./cfx Object\ 1
- Root Entry
x \x{01}Ole
x \x{03}PIC
x \x{03}META
x \x{01}CompObj
x \x{03}ObjInfo
x \x{01}Ole10Native
x \x{01}Ole10ItemName

The …Native file is the one we want. For reasons that I still don't understand you still have to delete the first four bytes from that file which in our case then reveals:

ice:~/ole>file $'\001'Ole10Native  
Ole% 0Native: data
ice:~/ole>dd if=$'\001'Ole10Native skip=4 bs=1 of=Fixed
7648+0 records in
7648+0 records out
7648 bytes transferred in 0.025733 secs (297203 bytes/sec)
ice:~/ole>file Fixed
Fixed: PC bitmap, Windows 3.x format, % 97 x 75 x 4

the relevant .bmp file. Yay!

– Sec

P.S.: If you have a stromg stomach, check the file format specification.

P.P.S.: In the meatime I found out that 7-Zip can also extract OLE Compund Files. Would've been a bit easier :-/

Ich bin ja nun schon seit einiger Zeit zufriedener Besitzer eines Android-Handys (ein G1). Durch einen bizarren Bug, und einen kleinen Fehler meinerseits habe ich meine Kontakt- und Kalender-Datenbanken verloren. [Ich synce meine Kontakte prinzipiell nicht mit Google].

Alle relevanten Datenbanken liegen bei Android alle unter /data/data.
(Nur wer sich root-Rechte auf seinem Telefon gesichert hat kann ab hier fröhlich mitspielen).

Die Kontakte sind unter com.android.providers.contacts/databases/contacts.db, die Termine unter com.android.providers.calendar/databases/calendar.db – Achtung: immer auf den richtigen Owner achten, ist ja ein Unixoides System. (Bei mir sind das momentan app_1 für die Kontakte und app_29 für den Kalender) – Bei Unklarheiten einfach nachschauen wem das .../databases Directory gehört.)

Mein letzte Backup dieser Files war ein nandroid-Backup - Die nandroid-Backups liegen alle auf der SD-Card unter /sdcard/nandroid/<handy-id>
Dort pickt man sich das gewünschte Subdirectory (meistens wohl das neueste) heraus, und darin das data.img. Das ist jetzt ein YAFFS2-Image das man z.b. mit unyaffs (Windows-Binary hier) auspacken kann.

Die passenden Files kann man nun aufs Handy verfrachten. Dabei empfiehlt es sich vorher nachzuschauen wer der richtige Owner ist, da adb push diesen leider überschreibt.

adb shell ls -l /data/data/com.android.providers.contacts/databases/contacts.db
adb push contacts.db /data/data/com.android.providers.contacts/databases/contacts.db
adb shell chown app_1:app_1 /data/data/com.android.providers.contacts/databases/contacts.db

adb shell ls -l /data/data/com.android.providers.calendar/databases/calendar.db
adb push calendar.db /data/data/com.android.providers.calendar/databases/calendar.db
adb shell chown app_29:app_29 /data/data/com.android.providers.calendar/databases/calendar.db

Damit die Applikationen die Änderung hinter ihrem Rücken auch mitbekommen empfiehlt sich entweder ein reboot, oder ein (brutaler) restart der passenden Dienste:

adb shell killall android.process.acore
adb shell killall com.android.calendar

Und schon sind die Daten wieder da. Yay!

– Sec

Als Nachtrag zu dem ersten Artikel.

Die Icons verschiedener Extensions im Statusbar wandern ja beim installieren neuer Extensions immer lustig hin und her. Im ersten Artikel hatte ich beschrieben wie man das von Hand reparieren kann.

Inzwischen geht das auch deutlich einfacher. Wie? Mit der Organise Status Bar Extension.

Yay, kein manuelles Gefummel mehr :-)

Und wenn ich schon mal über Firefox schreibe – meine aktuellen lieblings-Extensions:

• Adblock Plus
• Certificate Patrol
• CookieSafe
• Echofon
• Firebug
• FoxyProxy
• Ghostery
• Greasemonkey
• HTML Validator
• Lazarus
• Mouse Gestures Redox
• NoScript
• NoSquint
• Operator
• Organise Status Bar
• RefControl
• ScrapBook
• TabGroups Manager
• TabMixPlus
• Web Developer

Welche fehlen eurer Meinung nach?

– Sec

Many of you probably know the annoying behaviour of console-based programs like less or vi restoring the screen after exiting so that the text you just had there vanishes.
I never understood why you'd want that, and its commonly enabled by default on Linux machines.

The underlying feature is called alternate screen and basically its a feature of your terminal (be it xterm, gnome-term or even your console).

If you want to get rid of it, you have a few options:

Tell each and every program, not to use that feature.

A prime example is the less option -X.
This is useful as a quick ad-hoc solution, but not really feasible for everyday use - You don't really want to (re-)configure every program.

Tell your terminal not to enable this feature.

XTerm has an option located in the Ctrl+Middle mousebutton -Menu and is called Enable Alternate Screen Switching. Turn if off and all programs within will behave nicely.
If you want it for all your xterm windows, you can add something like this to your ~/.Xdefaults:

  XTerm*titeInhibit:  true
  UXTerm*titeInhibit: true

If you're using gnome-terminal, this is not for you. Gnome-terminal simply provides no way to disable this feature. Boo

Lie to your programs and say your terminal doesn't support it.

All (well-behaved) programs check if your terminal supports alternate screens before using them. This information is stored in the system terminfo or termcap database. Whether your system is using termcap or terminfo is an historical thing – Linux/SysV based systems tend to use terminfo, BSDish systems termcap.

This database has an entry for every terminal type your system supports. Your terminal is stored in the $TERM environment variable. Go and check your $TERM value. It is most probably xterm or something similar.

The capabilities for alternate screen are called ti and te in termcap and smcup / rmcup in terminfo. (Yay! for gratuitous differences). What we will be doing, is to override these by placing a suitably modified file in your $HOME.

For terminfo it works like this:

  infocmp -l $TERM > $TERM.src
  vi $TERM.src # remove the smcup= and rmcup= parts till the next ,
  mkdir ~/.terminfo
  tic $TERM.src

This dumps the current entry to a file, and compiles it into a file in ~/.terminfo/ after you modified it.

Termcap just uses plaintext files, so go look into your system termcap file (/usr/share/misc/termcap or similar), and find the entry for your current $TERM. On my system it looks something like this:

  xterm|xterm-color|X11 terminal emulator:\
          :ti@:te@:tc=xterm-xfree86:

copy that into ~/.termcap, and add :ti@:te@ at the beginning of the second line to disable these two capabilities. As you can see from my example on FreeBSD this is done by default for xterm and xterm-color.

That should fix it for you.

If you use screen

Lastly, if you are using screen, there is a simpler way to do it. Screen has a built-in feature to edit the termcap/terminfo for programs running inside. Simply add

  termcapinfo xterm* ti@:te@

to your ~/.screenrc (of course replace xterm with your $TERM in use before you start screen)

I hope this lengthy explanation did not bore you too much…

– Sec

In my ongoing quest to get rid of unnecessary use of WinWord, I moved a two-page checklist into our internal wiki. Unfortunately I came across two problems while doing that:

1. HTML <li>s don't support big empty circles. They default to black bullets, and the CSS {"list-style-type: disc;} only creates tiny circles.

   Unicode to the rescue: \x25EF, also known as \N{LARGE CIRCLE} or ◯ makes a fine replacement. Two lines of CSS to fix your list:

   ul { list-style-type:none; }
   ul li:before { content: "\25EF"; }
   

2. No page breaks. The list was on two pages for a reason and I needed to keep it that way. That was easily fixed by inserting

   <br class="break" /> 
   

   and making the break with CSS:

   br.break { page-break-before: always; }
   

   My first attempt was to use a <hr> instead – but strangely enough it made Firefox print another empty page at the beginning when used inside our wiki.
   

Putting it all together for TWiki:

<!-- <pre> -->
<style type="text/css" media="all">
br.pagebreak { page-break-before: always; }
.patternTopic > ul { list-style-type:none; }
.patternTopic > ul > li:before { content: "\25EF"; }
</style>
<!-- </pre> -->

The .patternTopic makes it only apply within the article, and the > makes it only apply on the first level after that.

– Sec

Last time we discussed the block order of linux-mdraid.

How do you find out which block order your RAID has?

The simplest way requires a working RAID to test against. (Ray created a small (50MB) test-RAID for that). First we get the first few blocks from each raw disk:

for $disk in sda1 sdb1 sdc1 sdd1 ;do
  for $nr in 0 1 2 3 4;do
    dd if=/dev/$disk skip=$nr count=1 bs=16k of=B.$disk.$nr
  done
done
dd if=/dev/md0 bs=16k count=20 of=RAID

Note that this assumes your stripe-size is 16k. If you know it is different, change it, if not you will find out later and have to retry with an adjusted value.

Now try to match up the first block with the RAID contents like this:

cat B.sda1.0 | cmp - RAID

If the block matches, you will get:

cmp: EOF on stdin

If the block is the wrong one:

stdin RAID differ: char 1, line 1

If none of your first blocks (the files ending with .0) match, either your block-size is too big (try again with half the previous size) or your RAID prefixes the disks with some internal bookkeeping info (in that case you can try to start with later blocks)

Now try to match the next blocks by adding then one by one to the cat command line like this:

cat B.sda1.0 B.sdb1.0 | cmp - RAID

that way you will easily recover the block allocation order of your RAID.

For example our linux-mdraid starts like this:

cat B.sda1.0 B.sdb1.0 B.sdc1.0 B.sdd1.1 B.sda1.1 ...

After that, its only a two-line patch to raidextract to fix that – hope you know C ;-)

That concludes our intermission for today. Tomorrow we will see why all this work wasn't even necessary.

– Sec

We left our heroes yesterday with a broken RAID 5 due to read errors on multiple disks. (Read part I: here)

A great starting point is raidextract by Peter Benie which attempts to re-assemble a RAID5. His web-page on this tool also serves as a great overview in the inner workings of standard RAID 5.

The first problem we stumbled upon is the fact that it assumes a certain pattern of the Parity blocks. (All examples from here on assume 4 disks, since that is what we had. But of course that's all applicable to any number of disks)

Adapting the example from his page:

Our Linux-mdraid unfortunately didn't conform to this expectation. Not only does it start with parity on the last disk (which raidextract would support with --rotate), but it also moves the parity block 'backward' instead of 'forward'.

The correct allocation order looks like this:

A quick&dirty hack to raidextract to implement this order:

--- raidextract.c	2008-07-26 11:33:53.000000000 +0200
+++ raidextract-new.c	2010-06-28 13:49:54.000000000 +0200
@@ -316,8 +316,10 @@
 	int paritydisk=(stripe / (disks-1) + rotate) % disks;
 	int len=stripesize-offset;
 	int bytes;
+	int ndisk;
 	char *ptr;
 
+	ndisk=(disk-paritydisk+3)%disks;
 	if (!noparity && paritydisk <= disk) disk++;
 	if (len>raidlen) len=raidlen;
 	if (winoffset+len > datasize) len=datasize-winoffset;
@@ -337,7 +339,7 @@
 	raidstart+=len;
 	raidlen-=len;
 
-	ptr=window[disk][windowalt]+winoffset;
+	ptr=window[ndisk][windowalt]+winoffset;
 	while (len)
 	{
 		bytes=write(STDOUT_FILENO, ptr, len);

A run on an error-free test-RAID confirms this and extracted it correctly. Yay!

continued in part III, coming soon

– Sec

The Munich CCC fileserver uses (as many other servers) software RAID 5 amongst its disks. We all (should) know that RAIDs are no substitute for backups, which was reinforced by a recent problem we had. While RAID level 5 can recover gracefully from a single failed disk, it generally can't cope with multiple failed disks at the same time.

One of the problems with large harddisks is, that there may be yet undetected errors on it, just because you haven't attempted to read that part for quite some time. Now when you start a rebuild of a RAID5, these errors quickly pop due to the rebuild process needing to read all the data. This is the main reason why you should regularly run complete surface scans on your RAID arrays.

Almost all RAID implementations tend to mark a whole disk as failed as soon as it contains a single error. This becomes a problem as soon as you detect a second error on your currently degraded RAID you are just attempting to rebuild.

Fortunately there is still hope. If the errors on your failing disks occur on non-overlapping points of the array, you can recover a complete copy of your data by assembling just the right pieces. But unfortunately there appears to be no hardware or software RAID solution able to do that out of the box. So we're left to try this manually.

more on this saga in part II, coming soon…

– Sec