Category: Uncategorized

  • Ubuntu 22.04 LTS (debian) Apache2 http2 + PHP 8.2

    [Updated 2023-07-21]
    Commands to launch and to set up my preferred environment in ubuntu.

    sudo su
    add-apt-repository ppa:ondrej/php --yes
    apt update && apt upgrade -y && apt autoremove -y
    
    # Install all php related components.
    apt install -y php8.2-fpm php8.2 libapache2-mod-php8.2 php8.2-common  php8.2-xml php8.2-xmlrpc php8.2-curl php8.2-gd php8.2-imagick php8.2-cli php8.2-imap php8.2-mbstring php8.2-opcache php8.2-soap php8.2-zip php8.2-intl php8.2-bcmath unzip php8.2-dom apache2 composer libapache2-mod-fcgid certbot python3-certbot-apache 
    
    apt autoremove -y
    

    Optionally to setup ZSH and the Z plugin

    $> sudo apt install -y zsh #optional for ZSH
    $> zsh
    #pick option #2, to setup.
    
    # install Z plugin
    $> curl https://raw.githubusercontent.com/agkozak/zsh-z/master/zsh-z.plugin.zsh > zsh-z.plugin.zsh
    $> echo source ~/zsh-z.plugin.zsh >> .zshrc
    $> source ~/zsh-z.plugin.zsh
    
    #set current user to use zsh as default
    $> sudo chsh -s $(which zsh) $(whoami)

    Setup apache.

    ### SETUP YOUR FQD - fully qualified domain
    $> DOMAIN_NAME=example.com

    Paste below carefully…

    sudo mkdir -p /var/www/$DOMAIN_NAME/public
    cd /var/www/$DOMAIN_NAME
    sudo chown -R $(whoami):www-data .
    
    
    sudo tee /etc/apache2/sites-available/$DOMAIN_NAME.conf << END
    <VirtualHost *:80>
        ServerName $DOMAIN_NAME
        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/$DOMAIN_NAME/public
        ErrorLog \${APACHE_LOG_DIR}/error.log
        CustomLog \${APACHE_LOG_DIR}/access.log combined
    
        Protocols h2 http/1.1
    
    	<Directory /var/www/$DOMAIN_NAME/public>
    		AllowOverride All
    		Order Allow,Deny
    		Allow from All
    		DirectoryIndex index.php
    	</Directory>
    </VirtualHost>
    END
    
    sudo a2dissite 000-default
    sudo a2ensite $DOMAIN_NAME.conf 
    sudo a2dismod php8 #may fail... fine.
    sudo a2dismod php8.0 #may fail... fine.
    sudo a2dismod php8.1 #may fail... fine.
    sudo a2dismod php8.2 #may fail... fine.
    sudo a2dismod mpm_prefork
    sudo a2enmod mpm_event
    sudo a2enmod http2
    sudo a2enmod proxy
    sudo a2enmod proxy_fcgi
    sudo a2enconf php8.2-fpm
    sudo a2enmod rewrite
    sudo systemctl restart apache2

    Probably want to ensure that your DNS settings are working. Run certbot, or put in your SSL certs manually.

    other tidbits of script
    
    ###
    ### WARNING, this command can be dangerous.
    ### REVIEW initial crontab -l first, and only if you know for suer
    ### run the command below, and will populate.
    ### Handy cron stuff... INITIAL install only... REVIEW
    ###
    echo '@daily apt update
    @daily apt autoremove -y' | sudo crontab
    
    
    
    
    ###
    ### MySQL support for php
    ###
    apt install php8.2-mysql mysql-server php8.2-pdo
    
    
    ###
    ### Postgresl SQL support for php
    ###
    
    apt install php8.2-pgsql
    sudo -i -u postgres
    createuser --interactive
    
    
    ### Postgresql itself
    apt install postgresql postgresql-contrib
    sudo systemctl start postgresql.service
    
    
    ### Restart apache:
    $> sudo systemctl reload apache2
    

    Troubleshooting:

    Ensure the firewall is open to ports 80/443

    apache2.conf:

    SetHandler application/x-httpd-php

  • Linux ssh troubleshooting

    ensure that the home directory of the user is 0700, and all sub files.

    $~> chmod 0700 /home/<USERNAME>"

    other useful commands

    $> ssh-copy-id <user>@<targetSystemAddress> # copies your ssh keys over to the new system
  • PHPStorm and xdebug links

    • phpStorm 2021.3 RC
    • PHP 8.1.0
    • xdebug 3.1.1

    I enjoy the fact that there’s was the ability to change xdebug.file_link_format parameter in php.ini to allow integration between xdebug messages, and your IDE.

    However, recently, that seems to have broken/changed. The phpstorm:// protocol no longer works.

    After much troubleshooting I have it working for projects that use a framework, such as Symfony.

    You will now need to utilize the framework.yml ide field,

    https://symfony.com/doc/current/reference/configuration/framework.html#ide

    and enter it as follows:

    1. Comment out the xdebug.file_link_format, as that no longer works…
    2. edit: /config/packages/framework.yaml
    framework:
        ide: 'jetbrains://php-storm/navigate/reference?project=%env(resolve:PROJECT_NAME)%&path=%%f:%%l'

    3. edit: /.env ensure that this matches your project name… otherwise jetbrains won’t be able to find the correct session/window.

    PROJECT_NAME="the name of the project"

    4. clear cache, and now your links should work.

    5. restart apache/nginx if required.

    Your Symfony project will allow you to click on filenames (when errors occur) and integrations back to phpStorm.

  • Sending emails with symfony+swiftmail, so that the messages are grouped by conversation

    Easy enough to send emails with Symfony + SymfonyMailer messages out. The issue was once received in a program like Outlook, how do you group them into the same conversation.

    Instead of flooding an inbox with a ton of emails, the emails would be grouped by conversation.

    To accomplish this:

     <?php
    
    // in the __constructor, autowire MailerInterface into $this->mailer
    
    $subject = "something";
    $somethingHashedThatRepresentsThisConversation = "someHash";
    
    $messageId = "[email protected]";
    
           $email = (new Email())->from('[email protected]')
                                  ->to($targetEmail)
                                  ->subject($subject)
                                  ->html($html)
                                  ->text(strip_tags($html));
    
            $headers = $email->getHeaders();
            $headers->addIdHeader('Message-ID', $messageId);  // Original Message ID, I've found reusing it workss.
            $headers->addIdHeader('References', $messageId);  // Original MEssage ID to group by... could add others... I've found that this can reference itself.
    
            $email->setHeaders($headers);
    
    
            $this->mailer->send($email);
    

    There you go.

    PHP 8.0.x, Symfony 5.3

  • dnsmasq on pihole for wildcards

    Say you’re a developer and you want all domains that end in “.test” to be redirected to 127.0.0.1

    in the past, we’d edit /etc/hosts file and call it a day.

    But now, with multiple web services, and code that’s shared between multiple sites, we may want something a little more robust.

    DNSMasq to the rescue. I won’t go into details on how to install it, as it’s automatic with pihole.

    I’m using Pi Hole FTL version 5.2. (Date as of writing is 2020-Nov-04)

    1. create a new file called /etc/dnsmasq.d/99-custom.conf

    $> sudo nano /etc/dnsmasq.d/99-custom.conf

    2. The contents of the file should be this single line

    address=/.test/127.0.0.1

    the “.test” is the wildcard notation. 127.0.0.1 is the address you want it to map to.

    3. Restart pihole FTL

    $> sudo service pihole-FTL restart

    Now, on your local machine, ping anything.TEST to make sure it’s working.

  • USB to serial UART interface – use a nodeMCU prototype board to flash a TYWE2S (ESP8285)

    I was trying to flash a TYWE2s chip ESP8285. Bought an FTDI FT232R chip off amazon, two for something like $15… not realizing that I could use a ESP8266 nodeMCU prototype board to do it.

    Such a simple hack. You may not even have noticed that there’s an “EN” pin. Well, there is… look at the pinout for the nodeMCU… when it’s free, the esp8266 is enabled. Short it to ground, and it’s disabled.

    Now that’s the key, disable the chip, but the onboard chipset that helps with prototyping are enabled. So… that means the the TX and RX on the prototype board are still active and available to repurpose.

    So, to flash an ESP8285 (TYWE2S), we’re going to do this:

    • DO NOT PLUG IN THE nodeMCU yet.
    nodeMCUTYWE2S
    TXTX
    RXRX
    3.3V3.3V
    GNDGND
    ENGPIO – 00
    • Now power up the nodeMCU by plugging in the USB.

    If you wired everything correctly, and in the same order for powering up. You’re going to be fine.

    esptool.py will flash the ESP8285 on the TYWE2s chip via the nodeMCU usb serial.

    This procedure should work for the TYWE3S chip as well… but I haven’t tested this. This general procedure will work for all chips…

    NOTE: TX to TX, and RX to RX are not swapped in this case, because we’re hijacking the prototype board… on the board itself, it’s already cross labelled.

    Note1: The LED on the nodeMCU will be disabled.

    Note2: $> esptool.py –port /dev/cu.usbserial-0001 read_flash 0x00000 0x100000 fwbackup.bin

    will backup the firmware currently on the chip.

    Note3: Tasmotizer was used as a GUI to upload the new firmware.

  • Testing your power outlets/receptacles at home.

    Had some worn outlets, house is 13 years old… so, turn off the breaker for that circuit, and replaced the unit.

    The issue was when I plugged one of those “outlet” testers in, it told me I had an open neutral.

    Well, WTF… I plugged it in the other side, and then a bunch of other outlets… ALL of them had an open neutral.

    I’m a worried… youtube what the open neutral actually means, and yeah, It’s something’s busted if that’s the report.

    The problem wasn’t my outlet, or wiring, it was the tester. I took the multimeter out… and tested the connections.

    HOT and Neutral = 120 VAC
    HOT and Ground = 120 VAC
    Neutral and Ground = 0VAC.

    So how is that an open neutral… it’s not. If it were, Neutral to Ground would also be 120VAC.

    The tester plug is bad, or a dead LED inside, and not the wiring.

    🙂

  • Converting files HEIC to png/jpg or MOV to png/jpg

    If you got one or two files, just use an online converter…

    however, if you have several hundred files you wish to convert, use imagemagick.

    You first need to install it… on OSX, use homebrew.

    $> brew install imagemagick.

    this will give you the “convert” command in the command prompt.

    Now, to do it manually from the command prompt, you’d go:

    $> convert input.HEIC output.png

    That will convert your HEIC and make a png version of it. Your original is left alone.

    Now, if you have a batch of them… that’s another story… The lovely wife chose to print out a few hundred photos… so I had to go over “live” photos (MOV), and HEIC images, to be sent to Costco photo lab… they don’t do MOV/HEIC… so I have to convert them first.

    wrote a quick script, php…

    <?php
    /** HEIC to png. */
    $extension = ".HEIC";
    $files = glob("*" . $extension);
    foreach($files as $filename){
    
    	$basename = basename($filename, $extension);
    
    	echo "converting $filename...";
    	`convert "{$filename}" "{$basename}.png"`;
    	echo PHP_EOL;
    }
    

    here’s one for MOV. Taking the first frame.

    <?php
    /** MOV to png, first frame */
    $extension = ".MOV";
    $files = glob("*" . $extension);
    foreach($files as $filename){
    
    	$basename = basename($filename, $extension);
    
    	echo "converting $filename...";
    	`convert "{$filename}"[1] "{$basename}.png"`;
    	echo PHP_EOL;
    
    
    }
    

  • The quest to go paperless – workflow – Work in progess

    I’ve had a duplex automatic document fed scanner for close to a decade. Original a FUJISU Snapscan M1500, which Fuijisu ended support for a few years ago.

    Friend helped me obtain a Panasonic scanner, same form factor, but newer, faster. Network based.

    My original workflow consided of:

    1. scan documents, which automatically converted to OCR sandwiched PDFs
    2. Hazel (noodlesoft) would run some heuristic rules over the documents and rename and move the PDF to their appropriate spots. Leaving unhandled ones in a folder for manual processing. Though still leaving some manual tasks to handles.

    Since using the panasonic, I lost the luxury of automatic OCR. Abbyfine reader does an okay job, but bulk jobs, it crashes a lot. Spent the money, not particularly fond of it. I’m on MAC, and didn’t want to spin up a windows machine to just do this with the Panasonic included software.

    So, I’m in the process of recreating my workflow, in a more streamlined manner, with a combination of Linux and mac software.

    1. The Panasonic scanner is capable of scanning the documents as multi-image TIFF files, and store them on an SFTP.
    2. On the SFTP hosting server itself use inotify / fswatch to trigger:
      1. pdfsandwich
      2. move a original to “handled”
      3. move ORCed to “ORCed”
      4. move failed to ORC to “Failed”
        1. email me, to notify me of an error.
    3. Run heuristic rules against the content of the files.
      1. Search for dates that match any of the patterns
        1. yyyy-mm-dd / mmm d yyyy,
        2. “bill”/”invoice”/”printed” date
        3. etc…
      2. Look for key IDs
        1. account #s
        2. customer #s
        3. These IDs would be known, and be renamed to their formats, or move to their corresponding folders.
      3. look for any dollar amounts
        1. Total due
        2. $\d+\.\d+ (eg $313.23)
        3. $\d+ ($123)
        4. may need to look for negative amounts as well.
      4. Update any google sheet/CSV to include details of importance.
        1. For tax purposes.
          1. cell phone bills, amounts, for
          2. gas receipts
          3. restaurant receipts
          4. etc…
        2. include scan date, filename, and amounts.
      5. If no matching heuristics found – email me.

    Stage 1, and 2 are complete… stage 3 is more involved… need to work out some sort of pattern, for processing.

  • platformio clion – automation of gzip encoded files as bin2hex encode header files.

    This works for me on OSX Catalina 10.15.4

    Summary: If you wanted to automated include files, you can use the script shown to help automatically gzip, and convert bin to hex string representation for inclusion into C/C++ headers. Automating the process.

    This post explains the process to take html/css/icon (what have you) compresses them to gzip, and converts them to c/c++ include files. This is extremely useful for ESP8266/ESP32 webservers.

    FOLDERDescription
    <project root>your project root
    +/data/temporary hold spot (making it compatible with spiffs/littlefs location, in case you want to use them later.
    +/workdata/where you actually edit/changes your working files.
    +/src/data/location of include files.
    +/src/html.hgenerated file, that updates with the script.

    The bash script does the following (<project_root>/makeBinaryIncludeFiles.sh):

    #!/usr/bin/env bash
    
    rm -rf data
    mkdir data
    rsync -a workdata/* data/
    gzip -r data/*
    
    rm src/html.h
    touch src/html.h
    
    mkdir -p src/data
    
    find data -type f -print0 | while IFS= read -r -d '' filename; do
       xxd -i $filename > src/data/${filename##*/}.h
       sed -i .bak 's/unsigned char/const char/g;
       s/\] \=/] PROGMEM =/g' src/data/${filename##*/}.h
       rm src/data/${filename##*/}.h.bak
       echo "#include \"data/${filename##*/}.h\"" >> src/html.h
    done

    Platformio doesn’t run bash scripts (or at least I noticed it didn’t like to work), but doesn’t have a problem with python. So I wrote a wrapper to run the bash script:

    #makeBinarywrapper.py
    import subprocess
    rc = subprocess.Popen(['bash', './makeBinaryIncludeFiles.sh'])

    Add the following line to your platform.ini

    #platformio.ini
    extra_scripts = pre:makeBinaryWrapper.py

    With that peices together… you have have HTML/CSS/JS files, being submitted with the following command:

    // ...
    #include <html.h>
    // ...
    
    
    void handleHtmlGZ(const char *encodingString, const char *gzString, const int size) {
        digitalWrite(led, 0);
    
        server.sendHeader("content-encoding", "gzip");
        server.sendHeader("Expires", "Mon, 1 Jan 2222 10:10:10 GMT");
        server.send_P(200, encodingString, gzString, size);
    
        digitalWrite(led, 1);
    }
    
    
    void setup(){
    .....
    
    server.on("/", []{
            handleHtmlGZ("text/html", data_index_html_gz, data_index_html_gz_len);
        });
        server.on("/css/bootstrap.min.css", []{
            handleHtmlGZ("text/css", data_css_bootstrap_min_css_gz, data_css_bootstrap_min_css_gz_len);
        });
    
        server.on("/favicon.ico", []{
            handleHtmlGZ("image/x-icon", data_favicon_ico_gz, data_favicon_ico_gz_len);
        });
    
    ....
    }
    
    

    Making your c/c++ code much cleaner, and separated.