Tag Archives: troubleshooting


For a while now, I’ve been having problems with DKIM. It wasn’t working. My logs always had the same error:

And I’d look for a fix but never find anything useful.

Today I decided to go through my mail quarantine folders. In them I found several emails from a friend who is having problems with spammers using his email address. None of them are going through his mail server, they’re all spoofed. We’ve compared our SPF records and they look right. So I went and looked up why I’m seeing all these mails.

Turns out that not all mail admins have set up their servers right to look at SPF and block. That was my problem.

So I went and found a howto for my operating system to fix SPF with my Mail Transfer Agent (MTA). The document, provided by my VPS hosting provider, had how to set up SPF, how to configure my MTA to quarntine emails that fail SPF, a DKIM walk through, a ADSP howto, and a DMARC howto, all on the same page.

First things first. I fixed the SPF inbound. Now it should do the stuff it needs to. Then I figured since I still had time, I’d go after the DKIM problem.

So I backed up my existing files and followed along. AND NOTHING WORKED!. Still the same problem. Heck even the same error message.

So an

later and I started completely fresh. Nothing old, not even the old backup files.

And it still didn’t work. sysctrl status -l opendkim.service and journalctrl -xe were not much help either. Neither one gave enough information on what was wrong.

I did some searching through the logs, and found that even after changing the port to a local socet for Milter it still couldn’t work. But this time I found that it couldn’t see the files, and searching the directory that local socket should be in, it wasn’t there. After much googling I found an old bug report for Debian (my OS of choice). If the socket and pid files were missing, do this:

And suddenly everything was working. I sent test emails to test services, and they seem to be working. At least they told me that everything works.

Then I went why not and set up the ADSP and DMARC stuff in DNS.

Really just happy to get past the problem where dkim isn’t working. Now to go finish clearing out the quarantine files.

Why I don’t have a lab

An industry mailing list I’m on recently had a conversation that started asking about Master Degrees but had some hiring managers chip in. They said a question they tend to ask is to have the candidate tell about their home lab.

I’ve been asked this question a few times in the past, and I’ve asked people this question in job interviews. I know it’s to find out what kind of passion the candidate has for the job, but I think it’s starting to become a bad question to ask.

Here is why I don’t have a home lab.

Continue reading

Raspberry Pi WPA_Supplicant setup taking way longer than it should

—- TL/DR —-
If you want to connect a raspberry pi to a hidden access point your wpa_supplicant.conf needs to have the following in the network statement.

Don’t put a ” or a ) in your ASCII PSK it causes problems. I couldn’t get it to work with the hex psk using wpa_passphrase but I broke the rules of troubleshooting making multiple changes at a time instead of one and resetting it.

—– End TL/DR —-

I got a new phone. Nexus 5x.  But this isn’t what this post is about. My SO got a new phone last August. It was an unplanned by after the last one went for a porta-john swim. Again. not what this is about.

A co-worker suggested setting up a NAS to back up the pictures to. Being a poor college student, yes still grad school isn’t cheap, that really isn’t an option. Then he said well if it’s just the phones, why not use one of your Raspberry Pis?

The problem is getting the wpa_suplicant.conf file talking. It has taken me 2 days. Mainly because I don’t mess with that file much.

it hated my passphrase because of the ” and the ) in the middle example:

I tried wpa_passphrase with the file, but it didn’t like that either. (although I kind of want to go back and test it again, in case I missed something.

Which I couldn’t find out until after I set the AP to broadcast. So after more digging I found that

scan_ssid=1 has to be in the config.

Sigh. This is taking way longer than need be, and I’d just wire it, but it’s going to not be near the cables, because of power.

It’s all about the pcaps baby

So my android phone as an interesting problem, granted it’s an S4, running not the latest build so I don’t know if that problem still exists. Apparently the way the default mail application is set up, it can’t sync the mailboxes unless the Sync button is turned on. But that doesn’t stop that the mail application from trying to sync on a schedule.

Continue reading

Why is useful documentation hard to find?

I just finished reading The Linux Journal’s “Geek’s Guide to Enterprise Monitoring Success“. It was good, talking about how to leverage the monitoring to work for the IT department in an organization. This also talked about some business problems you can face, which I’ve seen first hand. I’ve been in the “metrics from another group’s monitoring tools” meeting before. Trust me, you need to be sure of yourself and what you’re doing for the company before that happens. I’ve also seen monitoring systems destroyed because the wrong people had too much access and trying to  tune the system for their needs only.

For what it was, this was a good guide. From the title though, I expected something different.

Continue reading

Yet more with Fail2Ban

So yesterday, I thought I was all good on Fail2Ban today’s logcheck emails show there were still problems with Dovecot.

Continue reading

More Fail2Ban fun with Debian Stretch

Yesterday, going through email yesterday, mostly logcheck emails, I found that Apache wasn’t blocking the attackers. It was seeing them, but not adding the ip address to iptables block list.

The fix was setting up the maxretry it was set rather high, I moved it down to 1 like I had it in the past. I also adjusted the search time to 1 hour and the ban time to 7 days. I thought I was good, and didn’t give it a second thought.

Today, looking at the logcheck emails (really it’s a great IDS for system admins to get a view into their box), there are a lot of automated attacks on the mail server NOT BEING BLOCKED!!! It worked yesterday, there were even banned ip addresses in the chain.

After lots of digging, and several changes that didn’t work, I decided to go for the bad option.

Really the real reason was that Fail2Ban had been around for a while. Things changed, and I had a weird mishmash of configuration files. After the install I removed the files in the package that were not debian related, not sure why bsd; osx; or fedora are in the Debian package to start with.

Followed the local customization file directions creating jail.d/server-defaults.conf with apache-auth and dovecot in them. ssh is handled by defaults-debian.conf. Why the new file, in case the Debian one gets over-written by new stuff later.

Restart the service and…

Still not working for dovecot.!? (tailing the log and watching iptables).

Turns out, it’s where Fail2Ban was set for default to watch for login errors for Dovecot (also noted through the logs). It’s looking in /var/log/mail.warn. I don’t know if I changed it, or it’s legacy left over, or what, but my box it’s /var/log/auth.log where Dovecot login failures go. So I added the logpath to jail.d/server-defaults.conf, restarted Fail2Ban and it worked.

Fail2Ban problems with Debian Stretch

This week The Debian project released “Jessie” (Debian 8.0) as stable. I like to keep my servers a little more ahead of the curve than that, so I upgraded to the new testing branch “Stretch”.

While going through my logs from yesterday and this morning, log checker is awesome, I saw someone hitting my mail server. Normally you only get 1 chance to log in as a non-existent account before Fail2ban kicks in and adds the ip address to my Netfilter iptables jail. This address kept showing up, hour after hour in the logs, and multiple user names.

Looking, I found out that while running, it wasn’t catching all the rules for Fail2ban. I checked the configuration files, and things checked out OK. So I fell back on the old restart the service and see what errors pop.

Continue reading