________________________________________
Let's start with a moderately simple regex:
/\A(?:\d{5})(?:-(?:\d{4}))?\z/
Some of you might be smacking your forehead at the thought of this being "simple", but bear with me. Let's break down what it does piece by piece. Match five digits, then optionally, a dash followed by four digits. All in non-capturing groups, and anchor to the beginning and end of the line. That tells you what it does, but not what it's for.
Explaining out the details in plain English, as in the above paragraph, doesn't help anyone understand what it's for. What you can do to help is have good variable naming and commenting, such as:
# Matches US zip codes with optional extensions
my $us_zip_code_re = qr/\A(?:\d{5})(?:-(?:\d{4}))?\z/;
Like any other code, we hand off contextual clues about its purpose using variable naming and comments. In Perl, qr// gives you a precompiled regex that you can carry around and match like:
if( $incoming_data =~ $us_zip_code_re ) { ... }
Which some languages handle by having a Regex object that you can put in a variable.
There are various proposals for improved syntax, but a different syntax wouldn't help with this more essential complexity. It could help with readability overall. Except that Perl implemented a feature for that a long time ago that doesn't so drastically change the approach: the /x modifier. It lets you put in whitespace and comments, which means you can indent things:
my $us_zip_code_re = qr/\A
(?:
\d{5} # First five digits required
)
(?:
# Dash and next four digits are optional
-
(?:
\d{4}
)
)?
\z/x;
Which admittedly still isn't perfect, but gives you hope of being maintainable. Your eyes don't immediately get lost in the punctuation.
I've used arrays and join() to implement a similar style in other languages, but it isn't quite the same:
let us_zip_code_re = [
"\A",
"(?:",
"\d{5}", // First five digits required
")",
"(?:",
// Dash and next four digits are optional
"-",
"(?:",
"\d{4}",
")",
")?",
].join( '' );
Which helps, but autoidenting text editors try to be smart with it and fail. Perl having the // syntax for regexes also means editors can handle syntax highlighting inside the regex, which doesn't work when it's just a bunch of strings.
More languages should implement the /x modifier.
________________________________________
I haven't updated the blog in a while, and I'm also rethinking the use of WordPress. So I decided to dump the old posts, and convert it to gemtext, the Gemini version of markdown.
The blog will be hosted on Gemini, as well as static HTTP.
Not all the old posts converted cleanly. A lot of the code examples don't come through with gemtext's preformatted blocks. Some embedded YouTube vids need their iframes converted to links. Comments are all tossed, which is no big loss since 95% of them were spam, anyway.
________________________________________
________________________________________
I slapped GStreamer1[1] together some years ago using the introspection bindings in Glib. Basically, you point Glib to the right file for the bindings, and it does most of the linking to the C library for you. They worked well enough for my project, so I put them up on CPAN. They cover gstreamer 1.0, as opposed to the GStreamer module, which covers 0.10 (which is a deprecated version).
It has an official Debian repository now, but I was informed recently that it would be removed if development continued to be inactive. Besides having enough time, I also don't feel totally qualified to keep it under my name. I don't know Gtk all that well, and don't have any other reason to do so anytime soon. I was just the guy who made some minimal effort and threw it up on CPAN.
I've found GStreamer to be a handy way to access the camera data in Perl on the Raspberry Pi. It's a bit of a dependency nightmare, and keeping it in the Debian apt-get repository makes installation a lot easier and faster.
So it's time to pass it on. As I mentioned, they're mostly introspection bindings at this point. It would be good to add a few custom bindings for things the introspection bindings don't cover (enums, I think), which would round out the whole feature set.
________________________________________
If you ask for advice on how to store passwords, you'll get some responses that are sensible enough. Don't roll your own crypto. Salted hashes aren't good enough against GPU attacks. Use a function with a configurable parameter that causes exponential growth in computation, like bcrypt. If the posters are real go-getters that day, they might even bring up timing attacks.
This is fine advice as far as it goes, but I think it's missing a big picture item: the advice has changed a number of times over the years[1], and we're probably not at the final answer yet. When people see the current standard advice, they tend to write systems that can't be easily changed. This is part of the reason why you still have companies using `crypt()` or unsalted MD5 for their passwords, approaches that are two or three generations of advice out of date. Someone has to flag it and then spend the effort needed for a migration. Having run such a migration myself, I've seen it reveal some thorny internal issues along the way.
Consider these situations:
Castellated is an approach for Node.js, written in Typescript, that allows password storage to be easily migrated to new methods. Out of the box, it supports encoding in argon2, bcrypt, scrypt, and plaintext (which is there mostly to assist testing). A password encoded this way will look something like this:
ca571e-v1-bcrypt-10-$2b$10$wOWIkiks.tbbftwkJ81BNeuOtq631SzbsVOO7VAHf5ziH.edAAqJi
This stores the encryption type, its parameters, and the encoded string. We pull this string out of a database, check that the user's password matches, and then see if the encoding is the preferred type. Arguments (such as the cost parameter) are also checked. If these factors don't match up, then we reencode the password with the new preferred types and store the result. This means that every time a user logs in (meaning we have the plaintext password available to us), the system automatically switches over to the preferred type.
There is also a fallback method which will help in migrating existing systems.
All matching is done using a linear-time algorithm to prevent timing attacks.
It's all up on npm now[2].
________________________________________
A common issue with makerspaces is letting people in the door. Members sign up and should have access immediately. Members leave and should have their access dropped just as fast. Given the popularity of the Raspberry Pi among makers, it makes sense to start there, but how do you handle the software end?
Doorbot.ts[1] is a modular solution to this problem. It's split into three parts:
A basic setup might read a keyfob from the reader, authenticate against an API held on a central server, and then (assuming everything checks out) activate a GPIO pin on the Raspberry Pi for 30 seconds. This pin could be connected to a solenoid or magnetic hold to unlock the door (usually with a MOSFET or relay, since the RPi can't drive the voltages for either of those).
The parts that interface to a Raspberry Pi (such as a Wiegand reader or activating a GPIO pin) are in a separate repository here: https://github.com/frezik/rpi-doorbot-ts[4]
The system can be compromised in quite a few ways. RFID tags can be copied. Cheap 10 digit keys can be brute forced. If you buy cheaply in bulk, there's a good chance those RFIDs are sequential or otherwise predictable. The Raspberry Pi can be compromised. The database can be compromised.
Which is not to say it's hopeless. That is, it's no more hopeless than what security you already have. An attacker can put a rock through any window. Physical locks can be picked, often without much skill at all. If you're like many makerspaces, you'll give a keyfob to anyone who shows up with membership dues, and you probably don't even check their ID.
So no, it's not the most secure system in the world, but it's also not the weakest point in the chain, either.
This isn't to say security should be ignored. Take basic precautions here, like using good passwords, and keeping the systems up to date.
Wiegand is a common protocol for RFID readers. There are cheap readers that work like a USB keyboard, "typing" the numbers of the fob. However, they're usually not built for outdoor mounting. There are cheap Wiegand options that are.
The protocol runs over two wires (plus power/ground). There is a Reader module in rpi-doorbot-ts for handling it directly. I don't recommend using it. The issue is that reading the data over GPIO pins means using tight timing, and doing this in Node.js is often unreliable. It often resulted in needing to scan fobs two or three times before getting a clean read.
Instead, you can use a C program[5] who's only job is to read Wiegand, interpret the bits, and spit out the number. Doorbot.ts then has as Reader that can read a filehandle. So you launch the C program, pipe its stdout into the Reader, and there you go. From practical testing, this has been far more reliable in scanning fobs the first time.
I recommend creating a directory in your home dir on your Raspberry Pi:
$ mkdir doorbot-deployment $ cd doorbot-deployment
And then create a basic `package.json`:
{ "name": "doorbot", "version": "1.0.0", "description": "", "main": "index.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1" }, "author": "", "license": "ISC", "dependencies": { "@frezik/doorbot-ts": "^0.13.0", "@frezik/rpi-doorbot-ts": "^0.6.0" } }
Be sure typescript is installed globally, and then install all the dependencies:
$ sudo npm install -g typescript $ npm install
Next, make an `index.ts` to run the system:
import * as Doorbot from '../index';
const INPUT_FILE = "database.json";
let reader = new Doorbot.FHReader( process.stdin ); let auth = new Doorbot.JSONAuthenticator( INPUT_FILE ); let act = new Doorbot.DoNothingActivator( () => { console.log( "Activate!" ); });
Doorbot.init_logger(); Doorbot.log.info( "Init" ); reader.setAuthenticator( auth ); auth.setActivator( act ); reader.init();
Doorbot.log.info( "Ready to read" ); reader .run() .then( () => {} );
This code creates a reader that works from STDIN, and authenticates against a JSON file, which would have entries like this:
{ "1234": true, "5678": true, "0123": false }
It then hooks the reader, authenticator, and activator together and starts it running. You can type the key codes ("1234", "5678", etc.) in to get a response.
I'll be following up with some more advanced usages in future posts.
________________________________________
TAP started as a simple way to test the Perl interpreter. It worked by outputting a test count on the first line, followed by a series of "ok" and "not ok" strings on subsequent lines. The hash character could be used for comments.
1..4 ok ok not ok # Better check if the frobniscator is working ok
There's been some additions over the years, but a lot of output of individual tests still looks a lot like this. Multiple tests could be wrapped together with `prove`. Here's what it looks like to run that on `Graphics::GVG`:
$ prove -I lib t/* t/001_load.t ............. ok t/002_pod.t .............. ok t/010_single_line.t ...... ok t/020_many_lines.t ....... ok t/030_circle.t ........... ok t/040_rect.t ............. ok t/050_color_variable.t ... ok t/060_num_variable.t ..... ok t/070_ellipse.t .......... ok t/080_regular_polygon.t .. ok t/090_int_variable.t ..... ok t/100_glow_effect.t ...... ok t/110_comments.t ......... ok t/120_point.t ............ ok t/130_include.t .......... skipped: Implement include files t/140_include_vars.t ..... skipped: Implement include files t/150_block_var.t ........ ok t/160_poly_offset.t ...... ok t/170_ast_to_string.t .... ok t/180_two_parses.t ....... ok t/190_meta.t ............. ok t/200_renderer.t ......... ok t/210_two_meta.t ......... ok t/220_named_params.t ..... ok t/230_renderer_class.t ... ok All tests successful. Files=25, Tests=91, 10 wallclock secs ( 0.10 usr 0.05 sys + 8.86 cusr 0.57 csys = 9.58 CPU) Result: PASS
Since TAP is just text output, it's easy to implement libraries for other languages. With `prove`'s `--exec` argument, we just pass an interpreter. Here's an example of a TypeScript project of mine:
$ prove --exec ts-node test/* test/activator_do_nothing.ts ........ ok test/activator_multi.ts ............. ok test/authenticator_always_false.ts .. ok test/authenticator_always.ts ........ ok test/authenticator_multi_fails.ts ... ok test/authenticator_multi.ts ......... ok test/logger.ts ...................... ok test/reader_fh.ts ................... ok test/reader_mock.ts ................. ok test/sanity.ts ...................... ok All tests successful. Files=10, Tests=12, 17 wallclock secs ( 0.05 usr 0.01 sys + 30.40 cusr 1.16 csys = 31.62 CPU) Result: PASS
Although the TAP package for JavaScript has a nice little runner all its own, which includes an automatic coverage report:
$ node_modules/.bin/tap --ts test/**/*.ts PASS test/activator_do_nothing.ts 1 OK 5s PASS test/authenticator_always_false.ts 1 OK 5s PASS test/activator_multi.ts 2 OK 5s PASS test/authenticator_always.ts 1 OK 5s PASS test/authenticator_multi_fails.ts 1 OK 5s PASS test/authenticator_multi.ts 1 OK 5s PASS test/logger.ts 1 OK 5s PASS test/reader_fh.ts 2 OK 5s PASS test/sanity.ts 1 OK 3s PASS test/reader_mock.ts 1 OK 3s
🌈 SUMMARY RESULTS 🌈
Suites: 10 passed, 10 of 10 completed Asserts: 12 passed, of 12 Time: 14s --------------------------|----------|----------|----------|----------|-------------------| File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s | --------------------------|----------|----------|----------|----------|-------------------| All files | 93.04 | 78.57 | 82.86 | 93.33 | | doorbot.ts | 100 | 80 | 100 | 100 | | index.ts | 100 | 80 | 100 | 100 | 26 | doorbot.ts/src | 91.4 | 77.78 | 81.82 | 92.31 | | activator_do_nothing.ts | 100 | 100 | 100 | 100 | | activator_multi.ts | 100 | 100 | 100 | 100 | | authenticator_always.ts | 100 | 100 | 100 | 100 | | authenticator_multi.ts | 100 | 80 | 100 | 100 | 22 | read_data.ts | 100 | 100 | 100 | 100 | | reader.ts | 28.57 | 100 | 25 | 33.33 | 39,40,41,44 | reader_fh.ts | 81.25 | 0 | 50 | 81.25 | 43,53,54 | reader_mock.ts | 100 | 100 | 100 | 100 | | --------------------------|----------|----------|----------|----------|-------------------|
Which, of course, can be dropped into the `scripts -> test` section of your package.json.
There's a feature of TAP that's been falling out of use in recent years, even in Perl, and that's test counts. Modern TAP allows you to put the test counts at the end, which can be done in `Test::More` by calling `done_testing()`. In the tap module for Node.js, you can simply not set a test count at all and it does it for you. Here's an example of that with a sanity test, which just makes sure we can run the test suite at all:
import * as tap from 'tap'; import * as doorbot from '../index';
tap.pass( "Things basically work" );
Avoiding a test count seems to be the trend in Perl modules these days. After all, automated test libraries in other languages don't have anything similar, and they seem to get by fine. If the test fails in the middle, that can be detected by a non-zero exit code. It's always felt like annoying bookkeeping, so why bother?
For simple tests like the above, I think that's fine. Failing with a non-zero exit code has worked reliably for me in the past.
However, there's one place where I think TAP had the right idea way back in 1988: event driven or otherwise asynchronous code. Systems like this have been popping up in Perl over the years, but naturally, it's Node.js that has built an entire ecosystem around the concept. Here's one of my tests that uses a callback system:
import * as tap from 'tap'; import * as doorbot from '../index'; import * as os from 'os';
doorbot.init_logger( os.tmpdir() + "/doorbot_test.log" );
tap.plan( 1 );
const always = new doorbot.AlwaysAuthenticator(); const act = new doorbot.DoNothingActivator( () => { tap.pass( "Callback made" ); }); always.setActivator( act );
const data = new doorbot.ReadData( "foo" ); const auth_promise = always.authenticate( data );
auth_promise.then( (res) => {} );
If the callback to run `tap.pass()` never gets hit, this test will fail. If you had removed the call to `tap.plan( 1 )` towards the start, it would pass as long as it compiles and doesn't otherwise hit a fatal error. Which is a pretty big thing to miss, since the callback is critical to the functionality of this particular module. Even if I knew it worked now, it might not in regression testing later.
Most (all?) other automated test frameworks around JavaScript have this problem. Sometimes, you can get around it by writing in clever ways, but it is far too easy to write test code that can erroneously declare success. Besides, shouldn't your tests be written in the most obvious, straightforward way? TAP had the answer decades ago.
________________________________________
________________________________________
________________________________________
The Perl Raspberry Pi ebook campaign[1] is going great so far. As promised, here's another sample chapter, where we cover the lineup of Raspberry Pi models, as well as some tools you'll want/need.
--------------------------------------------------------------------------------
The Raspberry Pi has expanded into a range of devices, which may seem overwhelming at first. There's also the matter of which tools you'll need once you do end up requiring one.
The initial offerings were the Pi Model A and B in 2012. These were both based on the Broadcom BCM2835 System on a Chip. The Model A had 256MB of RAM for $25, while the Model B had 512MB for $35.
These models also standardized the layout for most future Pi's, including the location of the pin header and HDMI port. These models also include a composite video connector.
In 2014, the A+ and B+ models were released, with prices reduced to $20 and $25, respectively. They expanded the pin header, which has been used on other Pi releases ever since. The separate composite video connector was also removed, with its functionality still accessible through a 3.5mm connector. The A+ also shrunk its form factor.
Released in 2014, the Compute Module was designed for industrial use. Instead of external ports, it uses the same connector as DDR2 SODIMM RAM (but don't try plugging it into your laptop). These require a special development kit for programming.
The Raspberry Pi Foundation made headlines in 2015 when they included an entire computer bundled with a magazine. The Pi Zero had an MSRP of $5, or free if you bought the December issue of the MagPi magazine. It introduced a smaller form factor that's not much bigger than a stick of gum, and used the same BCM2835 used in the original Model A and B. In 2017, the Pi Zero W was released for $10 that included onboard WiFi and Bluetooth, just like the B3/B3+ (see below).
In early 2015, the Pi B2 was introduced with a new BCM2836 SoC. This upgraded the CPU from single to quad core, and from 700MHz to 900MHz. It also bumped the memory to 1GB of RAM. This would be followed in 2016 by the B3, which has a BMC2837, bringing the clockrate to 1.2GHz. It also gave you built-in WiFi and Bluetooth, which means you no longer needed a separate dongle to connect your Pi to the Internet.
Both the B2 and B3 had an MSRP of $35.
While we wait for the next big iteration, the B3+ was released. It pushed the clockrate further to 1.4GHz, and now includes dual-band 802.11ac WiFi. It's also the first Pi to have gigabit Ethernet. Following tradition, it's priced at $35.
The Raspberry Pi Foundation isn't sitting still with the B3+ at the top of the model lineup, and are hard at work on the next major revision. Hoped-for improvements are support for USB3, a faster CPU, and breaking the 1GB RAM limit. We'll have to wait and see.
The hardware projects in this book can be easily done on any of the more recent models--Zero (W), Pi2, or Pi3. The Compute Module requires an extra development kit, which you will probably want to avoid until you have a specific need and experience working with the more mainstream boards.
If you need to fit the project in a small space, run off of low power, or want the cheapest option available, the Zero or Zero W are your best options. If you want speed and don't need to worry about power usage, the Pi3 is the best option. If you find a Pi2 for cheap, that's a perfectly good option, as well.
When the Raspberry Pi was first released, many people saw that it used a micro USB power port, and tried to use their leftover cellphone chargers to power it.
This did not work.
Cellphone chargers are meant for charging cellphone batteries. Under load, their voltage may sag below the level needed to keep the Pi running. A charger rated for 500mA may not be enough when you add a WiFi dongle, a monitor, a keyboard, and a mouse. It might boot up, but you'll likely run into strange problems with no obvious fix due to undervoltage/undercurrent.
The CanaKit micro USB power adapter is the standard way to power the Pi. It's rated for 2500mA, which is plenty to run even the Pi3.
Most of the Pi lineup doesn't have built-in WiFi. If you buy the Pi3 or the Zero W, you won't need one. If you can use wired ethernet and have a Pi with a wired port (not the A(+), Zero (W), or Compute Module), then you can also skip a WiFi dongle.
Otherwise, there are a number of cheap USB dongles that will work with the Pi. Edimax sells a small nub adapter. If you need more signal, the EDUP adapter has an external antenna, and works well with the Pi.
The main storage system on the Pi is an SD card (the Pi3 requires a micro SD card). If you take a quick look at SD card speed classes, you may be tempted to buy the first Class 10 or U3 card you can find.
Before you do that, note that the speed classes on SD cards refers to the sequential writing speed. This is good if you were putting your SD card in a camera, since streaming 60 frames of 4k video every second takes a lot of sequential writing. Running an operating system off that SD card does a lot of random access, not sequential. In practice, you'll often find that a quality Class 4 card works better than a cheap Class 10.
Recently, the A1 and A2 performance classes were released. These specify a minimum number of random IO operations per second, which is exactly what we want for the Pi. These are worth looking into.
You should look at benchmarks to compare SD cards on the Pi. The Pi's own SD card access rate is admittedly limited, so you may see the Pi hit a bottleneck before the SD card itself is maxed out.
As for size, at least 8GB will do. You can get away with 4GB, but you'll find yourself limited at that size.
The Zero and Zero W have only one USB port for connecting external devices, and it's a micro port. To connect anything, you'll need a male micro to female USB A adapter. If you need to connect more than one device, then you'll also need a hub, preferably a powered one so that you're not drawing too much from the Pi itself.
When I was young, I bought a $15 Radio Shack stick soldering iron that plugged directly into the wall. I was always frustrated at getting bad solder joints that would never form into the nice little volcano-shaped mounds you usually see on PCBs. I dreaded soldering pins that were close together, because it was always a struggle to keep the solder from spilling over and bridging the connections together. I thought I was bad at soldering.
I eventually bought a nice soldering iron with a good temperature control unit. Turned out, I was pretty good at soldering, I just had the wrong tool from the start.
Where that old stick iron simply heated the element up with power right from the socket, a temperature controlled iron measures the heat of the tip and supplies power to keep it near a setting. They heat up much faster, as well.
The one that I bought is the Kendal 937D, and I still use it as my main iron. The Hakko FX-888D is one of the more popular models out there, though I'm not a fan. It's more expensive than the Kendal, and setting the temperature is more cumbersome.
Be sure to get solder meant for electronics. Plumbing solder can damage electronics components. Other than that, something around 0.8mm thickness will do fine.
Even with a good iron, I sometimes make mistakes. The Solder Sucker uses a plunger or a squeeze bottle to suck the molten solder away. The wick uses capillary action to do the same. I tend to prefer the sucker method, but this is personal opinion. Either option is cheap, so it doesn't hurt to have both around.
Not everything has to be soldered together. A solderless breadboard has pins connected in rows, which is nice for prototyping before laying everything down on a circuit board and making it permanent. Buy a few different sizes.
These connect the Pi and other devices together using either the pin headers or a solderless breadboard. You'll need male-to-male, male-to-female, and female-to-female connectors. These are often available in a variety pack, in a flat strip where the individual wires can be pulled apart, and often you find these wire variety packs bundled with a breadboard (mentioned above).
These four components are the bread and butter of electronics. You'll want a variety of all these. 100 Ohm resistors are especially needed, since this is a good value for protecting LEDs from pulling too much current. Other than that, assortment packs are readily available for all of these.
Modern electronics manufacturing has moved towards surface mount components, which can't be plugged into a breadboard directly. They need a specific footprint on a PCB.
Fortunately, two of the major hobbyist electronics suppliers, Sparkfun and Adafruit, sell breakout boards for many common components. These are a small PCB with a pin header that can be plugged into a breadboard, and from there into a Pi using jumper wires.
We're going to be using the TMP102 temperature sensor chip in this book. It's available in a breakout board from both Sparkfun and Adafruit. The chapter on GPS was tested using the Adafruit Ultimate GPS Breakout, though any GPS unit that puts out NEMA over a serial connection should work.
The chapter on taking pictures will be using the Pi camera. It uses a special connector on the Pi, and is generally higher quality than any USB webcam you could buy for the same price.
________________________________________