Blog

Had some worn outlets, house is 13 years old… so, turn off the breaker for that circuit, and replaced the unit.

The issue was when I plugged one of those “outlet” testers in, it told me I had an open neutral.

Well, WTF… I plugged it in the other side, and then a bunch of other outlets… ALL of them had an open neutral.

I’m a worried… youtube what the open neutral actually means, and yeah, It’s something’s busted if that’s the report.

The problem wasn’t my outlet, or wiring, it was the tester. I took the multimeter out… and tested the connections.

HOT and Neutral = 120 VAC
HOT and Ground = 120 VAC
Neutral and Ground = 0VAC.

So how is that an open neutral… it’s not. If it were, Neutral to Ground would also be 120VAC.

The tester plug is bad, or a dead LED inside, and not the wiring.

🙂

If you got one or two files, just use an online converter…

however, if you have several hundred files you wish to convert, use imagemagick.

You first need to install it… on OSX, use homebrew.

$> brew install imagemagick.

this will give you the “convert” command in the command prompt.

Now, to do it manually from the command prompt, you’d go:

$> convert input.HEIC output.png

That will convert your HEIC and make a png version of it. Your original is left alone.

Now, if you have a batch of them… that’s another story… The lovely wife chose to print out a few hundred photos… so I had to go over “live” photos (MOV), and HEIC images, to be sent to Costco photo lab… they don’t do MOV/HEIC… so I have to convert them first.

wrote a quick script, php…

<?php
/** HEIC to png. */
$extension = ".HEIC";
$files = glob("*" . $extension);
foreach($files as $filename){

	$basename = basename($filename, $extension);

	echo "converting $filename...";
	`convert "{$filename}" "{$basename}.png"`;
	echo PHP_EOL;
}

here’s one for MOV. Taking the first frame.

<?php
/** MOV to png, first frame */
$extension = ".MOV";
$files = glob("*" . $extension);
foreach($files as $filename){

	$basename = basename($filename, $extension);

	echo "converting $filename...";
	`convert "{$filename}"[1] "{$basename}.png"`;
	echo PHP_EOL;


}

I’ve had a duplex automatic document fed scanner for close to a decade. Original a FUJISU Snapscan M1500, which Fuijisu ended support for a few years ago.

Friend helped me obtain a Panasonic scanner, same form factor, but newer, faster. Network based.

My original workflow consided of:

1. scan documents, which automatically converted to OCR sandwiched PDFs
2. Hazel (noodlesoft) would run some heuristic rules over the documents and rename and move the PDF to their appropriate spots. Leaving unhandled ones in a folder for manual processing. Though still leaving some manual tasks to handles.

Since using the panasonic, I lost the luxury of automatic OCR. Abbyfine reader does an okay job, but bulk jobs, it crashes a lot. Spent the money, not particularly fond of it. I’m on MAC, and didn’t want to spin up a windows machine to just do this with the Panasonic included software.

So, I’m in the process of recreating my workflow, in a more streamlined manner, with a combination of Linux and mac software.

1. The Panasonic scanner is capable of scanning the documents as multi-image TIFF files, and store them on an SFTP.
2. On the SFTP hosting server itself use inotify / fswatch to trigger:
   1. pdfsandwich
   2. move a original to “handled”
   3. move ORCed to “ORCed”
   4. move failed to ORC to “Failed”
      1. email me, to notify me of an error.
3. Run heuristic rules against the content of the files.
   1. Search for dates that match any of the patterns
      1. yyyy-mm-dd / mmm d yyyy,
      2. “bill”/”invoice”/”printed” date
      3. etc…
   2. Look for key IDs
      1. account #s
      2. customer #s
      3. These IDs would be known, and be renamed to their formats, or move to their corresponding folders.
   3. look for any dollar amounts
      1. Total due
      2. $\d+\.\d+ (eg $313.23)
      3. $\d+ ($123)
      4. may need to look for negative amounts as well.
   4. Update any google sheet/CSV to include details of importance.
      1. For tax purposes.
         1. cell phone bills, amounts, for
         2. gas receipts
         3. restaurant receipts
         4. etc…
      2. include scan date, filename, and amounts.
   5. If no matching heuristics found – email me.

Stage 1, and 2 are complete… stage 3 is more involved… need to work out some sort of pattern, for processing.

This works for me on OSX Catalina 10.15.4

Summary: If you wanted to automated include files, you can use the script shown to help automatically gzip, and convert bin to hex string representation for inclusion into C/C++ headers. Automating the process.

This post explains the process to take html/css/icon (what have you) compresses them to gzip, and converts them to c/c++ include files. This is extremely useful for ESP8266/ESP32 webservers.

FOLDER	Description
<project root>	your project root
+/data/	temporary hold spot (making it compatible with spiffs/littlefs location, in case you want to use them later.
+/workdata/	where you actually edit/changes your working files.
+/src/data/	location of include files.
+/src/html.h	generated file, that updates with the script.

The bash script does the following (<project_root>/makeBinaryIncludeFiles.sh):

#!/usr/bin/env bash

rm -rf data
mkdir data
rsync -a workdata/* data/
gzip -r data/*

rm src/html.h
touch src/html.h

mkdir -p src/data

find data -type f -print0 | while IFS= read -r -d '' filename; do
   xxd -i $filename > src/data/${filename##*/}.h
   sed -i .bak 's/unsigned char/const char/g;
   s/\] \=/] PROGMEM =/g' src/data/${filename##*/}.h
   rm src/data/${filename##*/}.h.bak
   echo "#include \"data/${filename##*/}.h\"" >> src/html.h
done

Platformio doesn’t run bash scripts (or at least I noticed it didn’t like to work), but doesn’t have a problem with python. So I wrote a wrapper to run the bash script:

#makeBinarywrapper.py
import subprocess
rc = subprocess.Popen(['bash', './makeBinaryIncludeFiles.sh'])

Add the following line to your platform.ini

#platformio.ini
extra_scripts = pre:makeBinaryWrapper.py

With that peices together… you have have HTML/CSS/JS files, being submitted with the following command:

// ...
#include <html.h>
// ...


void handleHtmlGZ(const char *encodingString, const char *gzString, const int size) {
    digitalWrite(led, 0);

    server.sendHeader("content-encoding", "gzip");
    server.sendHeader("Expires", "Mon, 1 Jan 2222 10:10:10 GMT");
    server.send_P(200, encodingString, gzString, size);

    digitalWrite(led, 1);
}


void setup(){
.....

server.on("/", []{
        handleHtmlGZ("text/html", data_index_html_gz, data_index_html_gz_len);
    });
    server.on("/css/bootstrap.min.css", []{
        handleHtmlGZ("text/css", data_css_bootstrap_min_css_gz, data_css_bootstrap_min_css_gz_len);
    });

    server.on("/favicon.ico", []{
        handleHtmlGZ("image/x-icon", data_favicon_ico_gz, data_favicon_ico_gz_len);
    });

....
}

Making your c/c++ code much cleaner, and separated.

this was working as of 2020-05-23

apt update
apt install -y mosquitto mosquitto-clients
sudo systemctl enable mosquitto.service

Test the system to ensure it’s working. Start terminal windows…

term 1 – subscriber to the topic “test”

mosquitto_sub -t "test"

term 2 – publish something to the topic “test”

mosquitto_pub -m "message from mosquitto_pub client" -t "test"

Create a user/password

sudo mosquitto_passwd -c /etc/mosquitto/passwd username
Password: password

edit the file as root, /etc/mosquitto/conf.d/default.conf:

allow_anonymous false
password_file /etc/mosquitto/passwd

Restart

sudo systemctl restart mosquitto

test pub/sub in two terminals.

term-1 $> mosquitto_sub -t "test" -u "username" -P "password"
term-2 $> mosquitto_pub -t "test" -u "username" -P "password" -m "message from mosquitto_pub client"

That’s the setup of MQTT

Tasmota portion:

Tasmota is another beast… it wasn’t readily appearant to me when I was looking at the docs.

Tasmota 8.3.1

Enable MQTT support in tasmota:

1. Configuration -> Configure Other -> click MQTT Enable
2. Configuration -> Configuration MQTT
3. Host: the ip of the mqtt server
4. Port: should be 1833, unless you overrode it in the config.
5. Client leave as is.
6. User: the user you created for MQTT
7. Password: same password as you created earlier.
8. Topic: this should be unique for this device
9. Full Topic: %prefix%/%topic%/

Now, it wasn’t immediate appearant to me, what the mqtt topics where:

To turn on/off my device,

topic: "cmnd/aquriumlight/POWER"
payload: "on" / "off"

Groups are possible, the default group is “tasmotas”. The example below would turn on/off all devices listening to tasmotas.

topic: "cmnd/tasmotas/POWER"
payload: "on" / "off"

The device can be part of multiple groups:

issue the command in the console:

grouptopic
23:17:18 MQT: stat/aquariumlight/RESULT = {"GroupTopic1":"tasmotas","GroupTopic2":"","GroupTopic3":"","GroupTopic4":""}

GroupTopic1 is tasmotas, and topics 2-4 are empty.

to change that, in the console:

GroupTopic2 LivingRoom

sending a message to topic: “cmd/LivingRoom/Power” “off” would shut down all devices in LivingRoom.

to upgrade all devices (dangerous)

cmnd/tasmotas/upgrade

After you got the first tasmota devices setup, subsequent devices will be easy with mqtt.

MQTT Explorer

Using MQTT Explorer, it’s a GUI that will allow you easily send and view messages on the queue/topics. I recommend you check them out.

Connect to your mqtt server with the appropriate credentials.

Try out a few things:

EXMAPLE: Set your timezone on all devices to eastern daylight savings time. (GMT-4)
TOPIC: cmnd/tasmotas/timezone
PAYLOAD: -4

EXAMPLE: Set your timezone, and location on all devices to pacific, and location is Googleplex. (GMT-7)
TOPIC: cmnd/tasmotas/Backlog
PAYLOAD: timezone -7; latitude 37.422; longitude -122.084

I recently swapped servers, as my old one was running Ubuntu 14.04, I decided to just swap out the server completely to Ubuntu 20.04, port the data… hence the recent rsync post.

The process was simple:

• create new server
  • setup apache (vhosts), mysql, php, etc…
  • firewall rules should be the same, so you could use the same security group. A review should be done.
• rsync data over
• export database
• I did a smoke test, by editing my hosts file to point to the new server.
  • install software
  • restore data/backups.
• unbind/detach IP from old server
• repeat for new server.
• bind/attach old IP to new server
• Setup SSL certificates if you haven’t already.

Now, my old server, didn’t allow for IPv6… the new one did.

Create a new AAAA record, @/lloydleung.com to the IPv6 address.

I did have a mistake somewhere, where I created an “A” record for the www.lloydleung.com. I changed that to a C record, where www points to @.

Now when something tries to go to “www.lloydleung.com”, either IPv4 or IPv^, it looks up “lloydleung.com”, with the corresponding A or AAAA record.

Set your TTLs to something low, until you know it won’t change.

Many people now have Amazon echo/dots / Google home / Apple HomePod, smart TVs, smart thermostats, etc…

What happens if one of those get hacked? They’re already on your system? they’re authorized and allowed on your network… now what?

That’s why you should have a WiFi network dedicated to IoT devices on a separate VLAN. If one gets hacked, they’re not on your network.

You allow your main lan, to cross the VLAN without issue, but devices on the other side, no nothing outside their VLAN and their connections to the net (which some need to get updates, weather, etc).

Things that should be on your IoT lan: alexa/google/siri devices. your smart TV/android players/apple TVs/PS4/PS3/Xbox/etc… including your smart locks, and smart thermostats. Your media server should also be on that network.

Things to keep on your secured main:
your smartphone
laptops/computers
servers (NFS/backups)

Your cameras that don’t need ANY connection to the net could also be firewalled up on another VLAN, or as some others have done, put in firewall rules for those devices to not be allowed to connect anywhere, but allowed to be connected to from inside the lan (network security camera recorders).

Some common firewall rules…

your main LAN to IoT:
* all traffic

your IoT to Lan
* mDNS — this is so the devices can be discovered by the entire network.
* your devices that need connection back across the VLAN. Such as a network scanner, to sftp into a server to drop scanned images off
* mqtt traffic, because that’s what a lot of IoT devices do use for communications.
* ntp traffic out to the net.
* you could probably block out most other traffic… if you know you can.

Your IP Cameras should probably be firewalled to prevent anything going out, with the exception of “All Established and Related connections”.

It’s easiest to show what “All Established and Related connections” means. Say you’re devices behind your current router make a connection to a website, that website is able to send you back data — but that server couldn’t originally start/initiate communication with your browser.

This is not a simple, easy task to make and split your network. It will take several hours, and hardware that is compatible with it. My network ecosystem is 100% Ubiquiti. I did have a dlink smart switch, that stripped off my vlan, so non of the devices on the vlan could communicate out. After figuring that out, I moved all the devices off, and will retire that switch.

I think maybe still possible to use a switched port, that is dedicated to a VLAN, so all devices behind it will be on that VLAN… I’ll have to test that, but I’ve had enough networking stuff for the last few days.

TL;DR;

rsync -aHAXxv –numeric-ids –delete –info=progress2 -e “ssh -T -c [email protected] -o Compression=no -x” <source> <target>

options:

a – archive
H – preserve hard-links
A – preserve ACLS
X – preserve extend attributes
x – don’t cross filesystem boundaries
v – verbose
–numeric-ids – don’t map uid/gid values by user/group name
–delete – delete extraneous files from destination dirs
–info=progress2 — show progression
-e “ssh -T -c [email protected] -o Compression=no -x”

<source>
<target>

The flags made the sync much faster…

arcfour is dead, for a long time… same with blowfish… so [email protected] is the next best thing.

It’s been a while since I posted anything… Kids/family have taken up a lot of time.

Dylan’s started to ride two wheels, learning about the Pedrin method was a little too easy to teach him.

Dylan was in training wheels for a while and was ready, as he could ride a few meters without either wheel on the ground.

Every child is difference, your child is ready when they feel ready, so there’s no point in pushing it. Dylan (age 5.75), was ready but scared about falling. To put that at ease, it’s pretty simple.

You as the parent, stand at the front of the build, with the front wheel between your legs, to prevent the bike from falling.

Ask the child to balance, they won’t be able to, the point them experiencing how it feels before falling to either side. Give them a few tries, encourage them either way. The point is so they know what to do by putting their feet down.

Now… just give them a push, and away they go… my son was able to go a meter or two the first time, but then a house length, then two… he still needs practice, but this spring has been crazy with the weather dropping to 6°C from 20°C shorts weather.

[update: 2020-05-20] There does seem to be a conflict with Let’s Encrypt certbot. You’ll need to manually run updates. I’ll update once I figure it out, how to do it more automated. Looks like possible race condition for the ghost, and then the SSL… but since you’re defining them by macro’s it’s not so bad.

steps seem to be:

1. DNS to point to the server
2. Create vhost only (not SSL yet), and ensure it works.
3. Certbot certificate only
4. Add Use VhostSSL $domain line.
5. Renewals are certificates only…? <– untested

[Original Post] I have had to manage my share of vhosts, and it’s the same vhosts all the time. Maybe slight changes. They usually look something like this:

<VirtualHost *:80>
    ServerName domain.tld
    ServerAlias www.domain.tld

    DocumentRoot /var/www/project/public
    <Directory /var/www/project/public>
        AllowOverride All
        Order Allow,Deny
        Allow from All
    </Directory>

    ErrorLog /var/log/apache2/project_error.log
    CustomLog /var/log/apache2/project_access.log combined
</VirtualHost>

But repeated for EVERY domain name on the server. There’s an easier way, since apache 2.4.5.

You define the template macro:

<Macro VhostMacro $domain>
<VirtualHost *:80>
    ServerName $domain
    ServerAlias www.$domain

    DocumentRoot /var/www/$domain/public
    <Directory /var/www/$domain/public>
        AllowOverride All
        Order Allow,Deny
        Allow from All
    </Directory>

    ErrorLog /var/log/apache2/$domain_error.log
    CustomLog /var/log/apache2/$domain_access.log combined
</VirtualHost>
</Macro>

You call the macro:

Use VhostMacro lloydleung.com
Use VhostMacro site2.tld
Use VhostMacro something.com

You can extrapolate as much as you want.

Remember to restart/reload apache to load the new configs. Otherwise apache won’t know.