Archive for Uncategorized

Broken `apt-get update` Due To AppStream Error

Over this past weekend, I was trying to update my office desktop to the latest Ubuntu version, and hit a snag. It was giving an error on AppStream, so I tried to remove it using apt-get remove, but that, too, gave me errors, as you can see here:

Fetched 3,222 kB in 1s (1,836 kB/s)                           AppStream system cache was updated, but problems were found:
Metadata files have errors: /var/cache/app-info/xmls/fwupd.xml
Reading package lists... Done
E: Problem executing scripts APT::Update::Post-Invoke-Success 'if /usr/bin/test -w /var/cache/app-info -a -e /usr/bin/appstreamcli; then appstreamcli refresh-cache > /dev/null; fi'
E: Sub-process returned an error code
# apt-get remove libappstream3
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package libappstream3

A quick realization made me change libappstream3 to libappstream4, but that gave me a bit of a startling result:

# apt-get purge libappstream4
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  gnome-software-common lib32gcc1 libcuda1-384 libebur128-1 libegl1-mesa:i386 libgtkspell3-3-0 libidn11:i386 libpython3.6-minimal libsnapd-glib1 libstemmer0d libva-wayland1 nvidia-headless-390
  nvidia-opencl-icd-384 python3.6-minimal rename snapd-login-service xserver-xorg-legacy
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  appstream* gnome-software* gnome-software-plugin-snap* libappstream4* ubuntu-software*
0 upgraded, 0 newly installed, 5 to remove and 1932 not upgraded.
After this operation, 2,199 kB disk space will be freed.
Do you want to continue? [Y/n]

Whoa, there, champ. Hold your horses. I don’t feel like having my finally-working four-head Gnome display configuration getting hosed and having to do it all over again. So instead of removing, I tried reinstalling over top of the existing install:

# apt-get install --reinstall libappstream4
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  lib32gcc1 libcuda1-384 libebur128-1 libegl1-mesa:i386 libidn11:i386 libpython3.6-minimal libva-wayland1 nvidia-headless-390 nvidia-opencl-icd-384 python3.6-minimal rename xserver-xorg-legacy
Use 'apt autoremove' to remove them.
The following packages will be upgraded:
1 upgraded, 0 newly installed, 0 to remove and 1936 not upgraded.
Need to get 0 B/97.2 kB of archives.
After this operation, 33.8 kB of additional disk space will be used.
(Reading database ... 349307 files and directories currently installed.)
Preparing to unpack .../libappstream4_0.12.0-3ubuntu1_amd64.deb ...
Unpacking libappstream4:amd64 (0.12.0-3ubuntu1) over (0.10.6-2) ...
Setting up libappstream4:amd64 (0.12.0-3ubuntu1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...

Boom. Good to go. Now apt properly updates, and I can continue merrily on my way.

shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory

A few moments ago, while trying to restart the Postfix service on a client’s system, I was getting the following error:

shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory

Some strace runs later, I found that my pwd had been removed and recreated, so I was essentially in a state of limbo as far as the system was concerned. Quick fix:

cd && cd -

Brings me to my ~ directory, then back to the original pwd, and – most importantly – no more errors.

Custom sudo Login Prompt: Confuse Your Coworkers and Friends!

Quick way to have fun with Bash and sudo on a boring day. Insert the following into your /etc/bashrc, /etc/bash.bashrc, or similar file (as is appropriate for your distro and version):

alias sudo='sudo -p "Congratulations, %p! You are the one-millionth user to attempt to sudo to %U! Enter your password to see what you've won. "';

More info, from man sudo:

%H  expanded to the host name including the domain name (on if the machine's host name is fully qualified or the fqdn option is set in sudoers(5))
%h  expanded to the local host name without the domain name
%p  expanded to the name of the user whose password is being requested (respects the rootpw, targetpw and runaspw flags in sudoers(5))
%U  expanded to the login name of the user the command will be run as (defaults to root unless the -u option is also specified)
%u  expanded to the invoking user's login name
%%  two consecutive % characters are collapsed into a single % character
The prompt specified by the -p option will override the system password prompt on systems that support PAM unless the passprompt_override flag is disabled in sudoers.

2012 Redesign Proposal

Please view it here and send feedback to

PHP Saturation Survey

For the last few years, I’d wanted to write a spider that would crawl the web and gather some statistical information.  Specifically, I wanted to see how many sites reported being capable of serving PHP pages.  Finally, about two hours ago, I sat down to write the system that would collect this data, and within about 25 minutes, I had it doing my dirty work for me.  After I finished writing the core of the system, I then added a quick reporting tool to give me real-time statistics, because I was far too impatient to wait a few days, weeks, or even months before I could look at some numbers.

Keep in mind the following:

  • The spider gathers hostnames dynamically; it started by spidering the Google homepage (not search results), and went from there.
  • The recursion algorithm is not only imperfect — it’s impatient.  If a site doesn’t respond within 5 seconds, it’s omitted (uncommon).
  • Only homepages (indexes) are spidered.  This is still very effective, but will limit the results.
  • The recursion algorithm skips /^www[0-9]?/ subdomains, but will crawl others.
  • The system does not care if one site resides on the same physical or virtual server as any of the others.
  • The stats collection engine only gathers what the server is configured to tell it
  • We only check the HTTP headers; we do not spider the site itself to see if it links to any PHP scripts within itself.

This initial crawl will probably take at least a couple of weeks to reach some useful numbers.  Spidering sites to grab new URLs, check the database, ignore dupes, and recurse into the index to do it all again takes an average of 1.7 seconds per hostname.  Grabbing header data and recording statistics eats up about 3.0 seconds per host, and only works in chunks of hosts per run.

Since all you likely care about – much like myself – are the numbers themselves, you can find them here.  (Statistical data is no longer available.)  The text file will be updated there every five minutes.  If you’re interested in the latest figures and that file seems to be stale (i.e. – >10 minutes old), send me an email and I’ll go kick the crap out of the server and remind it who’s boss (until it becomes sentient, that is).

Over the next couple of days and weeks, I’ll spend a few more minutes here and there to improve the collection times, but the actual data itself is quite obviously of the most value in this project.  For now, all I care about is PHP market saturation, but I’m also collecting data on rival languages, web server software, cookies, page caching, OS and version information, and more.  Sure, other folks collect the same kind of information, but look at me…. do I look like them?  Then shut up.

Quick: How Many Potential Key Combinations Exist In 4096-bit Encryption?

Quite simple, glad you asked! That number is 24096. Or, in decimal format (forgive the lack of commas):

Einstein On The Moon – Tool: Sober

Getting dos2unix and unix2dos On Ubuntu

Since – unlike most other Linux distros – Ubuntu doesn’t come standard with either dos2unix or unix2dos, you have to install it yourself.  The package “tofrodos” in the repos is what you’ll want, and the simplest method to install it is, of course:

sudo apt-get install tofrodos

However, unless you want to retrain yourself to use their naming conventions (which I certainly don’t), you’ll want to add one final piece to the process:

vi ~/.bashrc

In there, add two new aliases:

alias dos2unix=’fromdos’;
alias unix2dos=’todos’;

Save and exit, and then simply type the following to update your session:

. ~/.bashrc

DO NOT, however, do that with your .bash_history file, as Philip Olson did a couple of years ago.  ;-P

Goodbye, s9y!

I’d finally had it.

Serendipity had a bunch of issues that were interruptive at best — destructive at worst.  I know plenty of other people have had success and been quite pleased with it, but it just wasn’t the right fit for me.  I’d known it for some time, but let’s face it: unless you’re a “blogger” by trade, you probably have more important things to do than update your personal hobby site.  I fell right into that camp myself.  I just couldn’t be bothered trying to convert it to something else.

Enter: WordPress.

Through one of my companies, PilotPig Web Hosting, we frequently get hosting orders for WordPress sites.  So the account is created, activated, and WordPress is installed for the customer.  I’ve done literally a few hundred installations and some basic customizations of the package, but I’ve never actually used it myself — until right now.  Seems decent.

Well, the one problem I thought I’d be facing was getting the posts from Serendipity imported into WordPress.  I was wrong.  All I needed to do was grab the full RSS 2.0 feed from s9y (/rss.php?version=2.0&all=1), save it to my Desktop, then run it through a quick conversion script that took all of two minutes to write:
if (!isset($argv[1])) {
echo 'Enter the RSS feed file to convert: ';
$argv[1] = trim(fread(STDIN,128));

$data = file_get_contents(dirname(__FILE__).DIRECTORY_SEPARATOR.$argv[1]);


Then just import it using the WordPress RSS importer tool, and – ta-da! – my work here was done, and I could go back to my day.  Neat.

Some Interesting Metrics on PHP-vs-ASP-vs-[etc.]

Due in part to the fact that our mailing list software rejects Google search URL’s as SPAM, and also because it’s potentially of interested to someone out there, here’s a copy of an email I tried to send to Richard Heyes on the PHP General mailing list regard market penetration and usage of PHP versus other languages.

Keep in mind that it was all typed into an email body, and is in it’s original form. It hasn’t been edited or cleaned up or anything, so don’t blame me if there are errors. Blame Richard. Why? Beats the hell out of me, just do it.

On Sun, Feb 8, 2009 at 09:35, Richard Heyes wrote:
> Hi,
> Can anyone point out some general statistics on PHP usage compared to
> other server languages? I’ve tried Netcraft, but they only appear (or
> I’ve only found) to have statistics on the httpd server used.


Use Google’s ‘filetype’ parameter. That’ll give you a general
idea of market penetration. It doesn’t give you any scientific stats
on the number of developers, nor does it account for files using
aliased processing or masking methods, but gives you a very basic idea
of the general shape and format of today’s World-Wide Web.

For this example, I’m only going to use *.com results (to maintain
some semblance of sanity here), and only currently-indexed files as of
this writing. I’m also only going to use extensions that make sense
(for example, you’ll find some .php7 files on Google, but we know
they’re not legitimate results). Also, understand that there will
most definitely be a margin of error, but considering the major
programming language developer’s preference for one language over
another, I’d say extension-spoofing will be minor enough to be
obscured by factual results. This is also pages indexed, not
sites indexed, nor servers reporting the languages as available.
Also, I’ve found that Google – for whatever reason – gives different
results when using cAsE-sEnSiTiViTy for `filetype` searches (though
the results themselves don’t appear to change), so in situations where
the numbers differ, only the HIGHEST number for that search is used.
Note the case in the search URL given.

Total .com results in Google:
18.610 Billion (100%)

(971M + 14.7M + 35.1M + 1.98M + 1.62M + 4,260) = 1.024B (5.5%)

(1.54B + 803M) = 2.343B (12.6%)

(300M + 22.1M + 5,450) = 322.105M (1.73%)

CGI + PL (We’re aware that .cgi can be anything, but we’ll couple
it with Perl here):
(89M + 41.3M) = 130.3M (0.7%)

RB (Other Ruby[-on-Rails] searches were too minute to mention):
231,000 (0.0012%)

PY (Python):
4.46M (0.024%)

HTML (for good measure):
2.29B (12.31%)

6.114B (32.85%)

Perhaps surprisingly, according to this, ASP is in the lead —
even surpassing plain HTML.

Disclaimer: By the time I send this email, the numbers will have
no doubt changed – and your results may also vary based upon your
geographic location due to Google’s network and data distribution.
This isn’t by any means scientific, yada-yada-yada, I’m not a
statistician or mathematician or even a nice guy, Google doesn’t
sponsor my work, blah, blah, blah. You know the drill.

And there you have it.