My not so uninteresting notes

Putting all together

mat — Sun, 19 Oct 2008 20:07:00 +0400

I have been busy last weeks making L4SUS more friendly or at least less complicated !

I am quite happy of what I acheived so far. As the title says I put everything together to make a real install guide and some documentation.

All of this is the 0.1 release you will need the zip and the tar file because it was easier to package one for windows and the other one for Unix/Linux/....

Enjoy !

l4sus_0.10.zip and l4sus_0.10.tar.gz

Fighting with call progress tones

mat — Sat, 11 Oct 2008 17:08:00 +0400

If you are like me a happy owner of any sippura (or now linksys spa) ata device and you are not living in US, then configuring the regional part of the device can be complicated. Especially the "Call Progress Tones" part.

First you have to know that the real important part is the following tones:

Dial Tone
Second Dial Tone
Outside Dial Tone
Prompt Tone
Busy Tone
Reorder Tone
Off Hook Warning Tone
Ring Back Tone

All this tones use a rule with the following syntax:

 freq1@db_level1,freq2@db_level2,...,freqn@db_leveln;
 num_seconds(frequency_sequence1,frequency_sequence2,...,frequency_sequencen)

With frequency_sequence with this syntax: num_seconds_on/num_seconds_off/frequencies

With this explainations 480@-19,620@-19;10(.5/.5/1+2) is quite simple to understand it is a signal of two frequencies :
480Hz and 620Hz both at -19dB during 10 seconds both frequencies (due to 1+2) will be played with this rhythm: half of second (.5) on and half of second off.

This more complicated one: 985@-16,1371@-16,1777@-16;*(.380/0/1,.274/0/2,.380/0/3,0/4/0) consists of 3 frequencies:
985Hz, 1371Hz and 1777Hz all at -16dB they will played forever (until status change for instance ...) with the following rhythm: 985Hz during 0,380 seconds then 1371Hz during 0,274 seconds then 1777Hz for 0,380 and finally no tone during 4 seconds.

If you are searching for the frequency for your country indications.conf from asterisk is your friend.

Fighting spam part 1: Spamtrap

mat — Thu, 04 Sep 2008 19:58:00 +0400

Why do we need to train the filter

Bayesian filters use a statistical approach to classify emails, in order to make it works you need to train the filter at the beginning with both know spam and not spam (ham) emails so that the filter knows which events are statistically present in spam emails and which are not. This is often done by the administrator (otherwise the bayesian part is not activated in most filters) but the day to day training is not so often done and not so well which leads to reduced filter efficiently as time goes by.

But in fact it's very important that the filter stays up to date with new spam messages so that it can gather new hints of spams and stays at the top. If the filter is not usually fed continuously with new spam messages it's because the task is not so easy.

So far I found 3 way to feed the filter with new spam messages:

Setup ham and spam folder for every users, fetch emails in each folder and inject them into the filter as new training email
Setup ham and spam email address, extract initial email from the message and inject it into the filter as new training email
Setup a spam trap so that all the email that go in will be injected in the filter for training.

The ham/spam folder solution

I won't talk much about this solution in this post, but this solution implies that you can easily read the email into this folder, IMAP server is then a necessity and emails have to be accessible. sa-learn-cyrus is a good script that helps you to do implement such kind of training.

The ham/spam mailbox

This solution shares lot of aspects with the previous one, it can be used even if you do not access to the raw email on server or if emails do not stay on server (ie. POP3 access). I will come back later in my series on a script to help to automate the analysis of emails sent by users. Basically you have to extract the real email and give it in input to the learning program of your filter (ie. sa-learn for spamassassin).

Problems with first 2 solutions

Each solution relies on the fact that users will classify emails, of such kind of behavior you can never be 100% sure. For misclassified spam (that is a valid message classified as spam) you can expect that you will get the information back because the user will be really annoyed that his email was wrongly classified that he will either put a copy in the dedicated directory or forward you to the appropriate mailbox.

But for the spam, people tends to just delete spam emails that are not classified (instead of forwarding them to the appropriate mailbox or moving it to the spam folder), and if they do it it won't be their priority so the training will be done later when it will not be so useful.

And last but not least problems is that users still continue to receive spam, most probably "bleeding edge" spams and so the filtering service is percepted as no so efficient which is sad !

The spam trap

The idea behind the spam trap is simple: setup a couple of mail address not used and that will not receive real emails and train the filter with those emails as new spams. One might ask which addresses should be used and it's a good question because you must be sure that the address will receive (enough) spam so that it will be useful.

At this task, the bad habits of spamers of trying any combination of letters and numbers might be of a great help to find addresses to trap spam. In deed if your mail server is well configured, it should have a list of valid recipient and if an email arrive for an invalid recipient it will (should?) be rejected by your server (if not I suggest that you start with this because it's a rather good practice).

In my case I use postfix and when an email is rejected I have the message : User unknown in local recipient table, for instance:

postfix/smtpd[32180]: NOQUEUE: reject: RCPT from unknown[85.102.177.38]:
  550 5.1.1 <64e4cf71d8d4f61c9f34e@matws.net>:
  Recipient address rejected: User unknown in local recipient table;
  from=<udp@neath-porttalbot.gov.uk>
  to=<64e4cf71d8d4f61c9f34e@matws.net>
  proto=ESMTP helo=<dsl85-102-45350.ttnet.net.tr>

So with the following command :

grep "User unknown in local recipient table" mail.log \
 | perl -ane 'm/ to=<([^@]+@[^>]+)> /; print "$1\n";'\
 | sort  | uniq -c | sort -n

I get a list like this:

     14 knfipen@matws.net
     18 2ec8fcca3ffcbb7cb960e@matws.net
     33 fixeq@matws.net
     45 lxyfxdybbh@matws.net
     48 gzxyvsf@matws.net
     65 kfiptmh@matws.net

So kfiptmh@matws.net or gzxyvsf@matws.net seems to be good target (or not so bad) for trapping spam. As we tend to receive a lot of spam once that our email is "discovered", you can expect to receive more spam shortly after starting accpeting email on a trap address which in our case is a good thing !

Now that we have identified potential addresses, you just have to instruct the mail server to send all the emails for those address to a script that will chunk them and train the filter. Here is a recipe when using postfix using the catch_spam script attached to this post

Adapt parameters

At the top of the script, adjust the parameters:

$host, host or ip address of the server running the spamd daemon, set to undef to desactivate this parameter (will use localhost as spamd server)
$user, Unix account for which the spam will be learned, set to undef to desactivate it (will use the .spamassassin directory of the user which started the script)
$dir_base, base directory holding new potential spam when not running in quiet mode, this directory must be writable by the user that will run the postfix service (ie. nobody)
$conf, a YAML configuration file (see example attached to this post) for filtering out some good email (based on regexp in header)

Create a new postfix service

We are about to create a catchspam service into postfix, this service will process all the emails caught by the trap, to do so add the following lines into /etc/postfix/master.cf:

catchspam unix  -       n       n       -       -       pipe
  user=nobody argv=/usr/local/script/catch_spam -q

Adapt:

argv so that it reflect the path where you installed the catch_spam script
user so that it reflect the Unix account with whom you learn spam (in which home directory bayes token are stored ...)

Add transport maps and tune delivery

In the /etc/postfix/main.cf add the following lines:

catchspam_destination_recipient_limit = 1
transport_maps = hash:/etc/postfix/transport

The latest line can already be in your /etc/postfix/main.cf, in this case there is no need to duplicate it.

Add the emails to valid recipient list

After settling on a list of emails that will be used for trapping spams, you need to add them to the list of valid recipient. Just add them to /etc/aliases with nobody as pair alias, ie:

# For kfiptmh@matws.net
kfiptmh: nobody
# For gzxyvsf@matws.net
gzxyvsf: nobody

Run newaliases, to make postfix aware of the change. Starting from now you are accepting emails for those trap addresses (gzxyvsf@matws.net and kfiptmh@matws.net in my example).

Create/Update /etc/postfix/transport

The latest step is to instruct postfix to forward emails for trap addresses via the postfix service created above, instead of trying to resolve the alias. We create or update /etc/postfix/transport by adding lines following the template bellow for each trap address

<trap_address> catchspam

For instance for my two trap addresses:

gzxyvsf@matws.net catchspam
kfiptmh@matws.net catchspam

Run the following command so that postfix can really use this file:

postmap /etc/postfix/transport

And finally restart postfix so that all the modification will be active in postfix.

Starting from now, the trap is working and any email send to the trap address will be automatically sent to spamassassin bayesian filter for spam training

Fighting spam part 0: Introduction

mat — Tue, 02 Sep 2008 15:42:00 +0400

I am about to write a few articles about not so bad technics to fight efficiently spam, along the past years I developped some technics to fight spam. The latest ones seems to provide a high ratio in term of efficiency it means high quantity of spam catched and almost no false positive. I started developping this for my own personnal domain and due to my current job expand and enhance this for the company where I work for.

At the beginning it was quite simple because for my personnal use, I work with thunderbird and it includes since a long time a very good spam filter which require not so much trainning before achieving a very good filter quality and so I didn't worried much about the quality of filtering done right on the server by the SPAM filter.

But, alas, thunderbird (as many other opensource project btw) is not corporate enougth and we are stuck with outlook ... The Junk filter of the latest is rather complicated and rather unusefull. So if you want to reduce the cries of the users about SPAM you have to find a good solution on the server.

The technics that I'll present are built around spamassassin and bayesian filtering, that's not revolutionnary technologies but with a fairly good (and not complicated) and quick tuning you can acheive a very good result.

It might seems unlogical (and it is a little bit) but I'll start this serie by an article on how to train automaticaly an already running spam filter based on bayesian filtering, article about how to setup it will follow but a bit later. My reason for this is that there is tons of guides on Internet on how to setup bayes in spamassassin, whereas articles on how to train it (without the help of the standard users feedback) are rare.

Part 1: setting a spamtrap

Microsoft my worst nightmare part 1.

mat — Mon, 23 Jun 2008 20:26:00 +0400

Intro

It is going to be a long story with a high number of sequels it seems.

I must confess that I do not have a high esteem for Microsoft products in general but my day work force me to use them or at least support user using it and more often than not I face real stupidity in the product.

Right now my key target is Outlook from Office 2003 edition.

Using Outlook, so remove Outlook Express ?

Well it might seems logical that if you know for sure that you will use Outlook then you won't need Outlook express. If you were ready like me to remove this component with image creation tools like nlite I should not recommend you to do so !

Why ? because if you do so and try to access to a IMAP/POP3 server, it will not work because you'll need registered DLL that comes with Outlook Express and could not be provided by Outlook. Sounds good ! I spend a few hours last week on this and find no way real way to escape ! (copying DLL failed because they need to be registered, copying + trying to register with regsrv32 failed on registration, reinstallation of Outlook failed). Don't get me wrong I didn't say that you can't manage is some situation to manage to register or install outlook (especially if you do not have installed security fixes but you might run into troubles.

There must be a good reason for this, and I can understand that maybe Microsoft guys wanted to mutualise code between two versions, cool good idea, but when you install Office if one or more DLL is missing then setup should install it and do what ever is needed so that thoses DLL will be installed and registered !

L4SUS update

mat — Sun, 22 Jun 2008 20:50:00 +0400

My first public release included some bugs.

I just release right now the version 0.02, it fixes lots of typo in my first release.

The whole system is pretty stable now, but it rely on automation both on the client and the server so now it's time to move on something else:

Create installer for windows using something like nsis so that simple configuration and scheduled tasks can be created automaticaly
See with OCS Ng what can be done to go further in the automation
Use offline database wsusscn2.cab to save bandwith when checking for updates (at least as an option)
Dig more deeply in the documentation of WSUS API in order to have more informations about updates.

The lastest version is here

To be continued ...

L4SUS

mat — Sun, 01 Jun 2008 21:04:00 +0400

From the theory to practice there is a huge gap use to say my teachers.

Well I faced them well trying for real the script described here in short every thing was mostly but in order to get something that is really exploitable more efforts were needed.

Now it's done and I packaged every thing in the zipfile attached to this post. This is mostly 3 scripts (and a few subscripts) bundled together, it require a samba server. I called this L4SUS and it stands for Linux For Server Update Service.

Using L4SUS should be quite simple:

Extract all .vbs script in a folder on each computer you want to manage updates
Rename updatelist.conf.example to updatelist.conf and adapt configuration (ie. the name of the samba server and the root path serving update)
Install perl script in the samba server, make it executable (chmod a+x) and adapt paths at the top of the script ($dest_base_dir and $update_file_dir)
Verify that Windows update is configurated to search (and only search) from udpates
On the samba server create a directory called files in the directory pointed by $dest_base_dir, and for each computer a directory of the name of the computer also is the directory pointed by $dest_base_dir

The most tricky part is that $dest_base_dir (in download_winupdate) must be exported as the value of filePath (in updatelist.conf).

Main components of this systems are :

getupdatelist.vbs, this script search for applicable updates on the computer where it is running, it create a file in its own folder called yyyymmddproposedupdate.log which contains a list of all updates, their id and their file's url.
download_winupdate, provide a computer name to this script and it will parse the latest proposedupdate.log file, then download missing updates files and create a command list:: upatelist which indicate to doupdates.vbs how to do the updates
doupdates.vbs, this script execute as most quietly as possible the different updates listed in updatelist

You can get a nearly automatic systems by scheduling the scripts via cron and windows scheduler. Of course in this case it does mostly the same as using directly windows update (well it should use less bandwidth but it seems not as clever as windows update when it comes to do all the updates quietly ...).

Expect more updates soon, because it still a bit rough and should need more polish.

Listing Windows Updates for fun ... and profit

mat — Sun, 18 May 2008 19:29:00 +0400

A couple of month ago I was searching for a solution for managing windows updates (and maybe more).

Out of the box you've got two solutions :

Standard Windows Update mechanisms
WSUS (Windows Server Update Service)

Both were not ok for my needs in a small sized company, here are the reasons:

Standard Windows Update

When you have non IT users (which is the case of nearly every companies) you must enable automatic updates.

Main drawbacks of this methods is that you don't control which updates are installed and which are not and each computers download a copy from internet which is inefficient and a pure waste of bandwidth and could even be a big problems when the size of company grow beyond a few tens of users.

WSUS

WSUS is a good solution from Microsoft to address the problems of standard update.

You setup the service and it will manage to find available updates, then you select the one you want and they will be downloaded. On the client you just have to change the address of the update server to point to your own update server and voilà everything is working !

But It oblige you to have a Windows 2000 or 2003 server and I really hate the strategy of lock down done by Microsoft.

As both solutions didn't suits my needs, I started looking for others. I found LSUS which is available into Samba-edu as it is an opensource project, I am pretty sure that it is quite easy to extract the LSUS part but I decided not to investigate more in this way.

At this moment I decided to investigate different solution and through Windows Update API manage to have something even not complete. The script listupdates.vbs is this result. This script for the moment just output the name and the url for the different updates, but it should not be very difficult to add the missing parts.