Rsync between Mac and Linux (Mac <->Linux)

 

Rsync between Mac and Linux

If you are of the small minority who use Rsync and have Mac and Linux computers you’ll want to read this.

I have Plex running on a Mac along with a second server, connected via IPSEC VPN, running Linux.  I have become increasingly annoyed with keeping these two servers in sync with the same data and the manual processes I use vary from operating system to operating system.  Over the years I've come up with a few scripts and processes that I use, but ultimately this just needs to be automated to the point that I don't have to worry about it other than the initial download.

My previous Rsync adventures, while somewhat basic in nature were always between two Linux machines, and once configured, I was able to add various functionality that improved my process.  However, it turns out the Mac and Linux machines I’m using have different ideas about the character sets their filenames are stored in. A file with an accented character on the Mac is completely different to one that looks the same on the Linux box.

The solution took a while for me to find but it’s very simple. Rsync has an option named --iconv to convert between character sets!

The solution was embarrassingly simple: Much due to a comment I read when researching the problem, I thought you were supposed to specify the character set in the order of transformation; but it seems as that is not the correct syntax. Rather, one should always use --iconv=utf-8-mac,utf-8 when initialising the rsync from the mac, and always use --iconv=utf-8,utf-8-mac when initialising the rsync from the linux machine, no matter if I want to sync files from the mac or linux machine.

Then it works like magic!

EDIT: Indeed, sometimes, checking the manual page closely is a good thing to do. Here it is, black on white:

1 2 3 4 5 6 7 8 9

--iconv=CONVERT_SPEC               Rsync  can  convert  filenames between character sets using this               option.  Using a CONVERT_SPEC of "." tells rsync to look up  the               default  character-set via the locale setting.  Alternately, you               can fully specify what conversion to do by giving a local and  a               remote   charset   separated   by   a   comma   in   the   order               --iconv=LOCAL,REMOTE, e.g.  --iconv=utf8,iso88591.   This  order               ensures  that the option will stay the same whether you're push-               ing  or  pulling  files.

My thanks to the random blogger who put the majority of this information out there...I've since lost where it came from so I can't give the necessary credit other than to say "well done".

Ubuntu Networking

 Today I had a system that for some reason was suddenly missing the network-manager and couldn't bring up the ethernet in the way I typically have.

The following was quick and simple and brought up the nic and requested dhcp.

sudo dhclient eth0  (or whatever your network adapter is)

If you need to look at your nic names, try looking in /etc/netplan/00-installer-config.yaml

CraftyController 4 - Ubuntu Install

CraftyController 4 Install on Ubuntu 22.04

I received the following error on Ubuntu 20.04 and 22.04, Post-install.  I had been following the documentation but later found that the instructions/links were for v3.  

Exception caught: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'  A virtual environment has been detected, would you like to try reinstalling the modules?[yes/no]: yes yes  Traceback (most recent call last): File "/var/opt/minecraft/crafty/crafty-web/crafty.py", line 21, in from OpenSSL import crypto, SSL File "/var/opt/minecraft/crafty/venv/lib/python3.10/site-packages/OpenSSL/init.py", line 8, in from OpenSSL import crypto, SSL File "/var/opt/minecraft/crafty/venv/lib/python3.10/site-packages/OpenSSL/crypto.py", line 1556, in class X509StoreFlags(object): File "/var/opt/minecraft/crafty/venv/lib/python3.10/site-packages/OpenSSL/crypto.py", line 1577, in X509StoreFlags CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK AttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/opt/minecraft/crafty/crafty-web/crafty.py", line 44, in with subprocess.Popen(command_list, stdout=subprocess.PIPE) as proc: File "/usr/lib/python3.10/subprocess.py", line 969, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.10/subprocess.py", line 1778, in _execute_child self.pid = _posixsubprocess.fork_exec( TypeError: expected str, bytes or os.PathLike object, not list

The following installed Version 4 and resolved the Post Install issues I was having with .v3

git clone https://gitlab.com/crafty-controller/crafty-installer-4.0.git && cd crafty-installer-4.0 && sudo ./install_crafty.sh 

Fortigate VM Config

 A little helpful tip to get an interface configured and able to route.

2. At the FortiManager VM login prompt, enter the username admin, then press Enter. By default, there is no password.
2. 
   
2. Using CLI commands, configure the port1 IP address and netmask.

For example:

config system interface

edit port1

set ip <IP address> <netmask>

end

4. 
   
4. 
   
4. To configure the default gateway, enter the following commands:

   config system route

   edit 1

   set device port1

   set gateway <gateway_ipv4_address>

   end

Find examples

 A few examples of using the "find" command to make your life easier.

My brother asked about searching through a folder to find images based on date range and then moving them to a new folder.  Sounds simple enough and I'm sure there are multiple ways to accomplish the same thing.  

As for me, I'll keep it simple and run the following little script.

#!/bin/bash
for i in $(find dir_to_search/ -newermt "2011-01-01" ! -newermt "2011-12-31"); do
mv $i /new_dir/
done

Another example of just doing the find for the date range:

find . -type f -newermt 2020-10-01 ! -newermt 2022-10-01

Fortigate -> Configure PiHole (like) DNS Setup

 FortiGate -> Pi Hole (like) Config 

I hate ads.

Fortigate has the ability to add external Block Lists and I determined to find lists and configure it to act much like PiHole.

Core Steps

• Add external filter lists

• Set up a DNS filter, block the external filter lists

• Set up a DNS server on the Fortigate with the DNS filter enabled
  
  

External lists

 Add the external lists. This is an example of 8 lists that are a mix of advertisers and trackers.

• SSH to Fortigate.
• At the CLI you can paste the following:

config system external-resource
    edit "Disconnect-Ad-Filter"
        set type domain
        set category 194
        set resource "https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt"
        set refresh-rate 60
    next
    edit "AdGuard"
        set type domain
        set category 192
        set resource "https://v.firebog.net/hosts/AdguardDNS.txt"
        set refresh-rate 60
    next
    edit "Firebog"
        set type domain
        set category 193
        set resource "https://v.firebog.net/hosts/Easylist.txt"
        set refresh-rate 60
    next
    edit "Firebog-Privacy"
        set type domain
        set category 195
        set resource "https://v.firebog.net/hosts/Easyprivacy.txt"
        set refresh-rate 60
    next
    edit "Fireblog-Admiral"
        set type domain
        set category 196
        set resource "https://v.firebog.net/hosts/Admiral.txt"
        set refresh-rate 60
    next
    edit "Fireblog-Privacy-2"
        set type domain
        set category 197
        set resource "https://v.firebog.net/hosts/Prigent-Ads.txt"
        set refresh-rate 60
    next
    edit "OISD"
        set type domain
        set category 198
        set resource "https://dbl.oisd.nl/"
        set refresh-rate 60
    next
    edit "OISD-IP"
        set type address
        set resource "https://hosts.oisd.nl/"
        set refresh-rate 60
    next
end
config system dns-server
end

DNS Filter

• Add a DNS filter. 
• You can do this under the 'Security Profiles' tab in the GUI of the Fortigate. 
• Enable the option FortiGuard Category Based Filter

• In the category filter list, locate an entry called 'Remote Categories'. Open, you will find seven of the external entries we added via CLI. 
•     Set all of them to 'Redirect to Block Portal'.

• At the bottom of the menu, you will find the option External IP Block lists. 
• Click at the + sign and select the entry 'OSID-IP'.

Now, all eight lists are active in this DNS filter profile.

Set up a DNS Server

Setting up a DNS server is most easy done by CLI (like most of things). Paste this code to your CLI to setup the server. Change the dnsfilter-profile to the name of your DNS profile.

    edit "internal"
        set mode forward-only
        set dnsfilter-profile "Your-DNS-Filter"
    next



After setting up your DNS server

• Publish it with your DHCP.

NOTE: This should get rid of many ads (YouTube ads cannot be blocked this way).

Samba - Test Config File

 You can easily test your Samba server for configuration errors. It check an smb.conf configuration file for internal correctness. If this program reports no problems, you can use the configuration file with confidence that smbd will successfully load the configuration file.

Open a shell prompt and then type the following command:# testparm
OR# testparm /etc/samba/smb.conf
We will see the following message on screen:
Load smb config files from /usr/local/etc/smb4.conf Loaded services file OK. Server role: ROLE_STANDALONE Press enter to see a dump of your service definitions

If you hit the [Enter], key you will see other details

OwnCloud Manual import of data

If you are here, you've obviously started to use Owncloud and found that importing the mountain of data you want accessible is painful. So what if you have gigabytes of files to add to your ownCloud server? You can of course use the desktop client to sync all those files, but what if we were talking about almost a terabyte?

Add files manually to your ownCloud server

This is probably the quickest and less complicated way of adding files. There are many alternative ways to transfer your files to your ownCloud server:

• FTP Upload;
• SCP Upload;
• Local disk to ownCloud disk transfer;
• USB disk to ownCloud disk transfer.

It's just as simple as dragging a folder to another folder. The only problem is that you might not know yet where ownCloud files are stored. I made life a little bit easier with this nifty little Bash snippet that automatically finds your ownCloud data directory:

find / -name config.php 2>/dev/null | while read line ; do cat "$line" | grep " 'datadirectory'" ; done

Your output should then look something like this:

So according to the snippet of code my ownCloud data directory is located at /var/www/owncloud/data so let's cd into that directory to check out what's in there.

As you can see, we have multiple files in here. My ownCloud username is bart, so let's take a look in that folder.

Now this looks interesting. If I check my ownCloud web portal I can see that the directory structure is the same:

So I created a folder called TestFolder123 in the web interface and executed ls -la on the server's console.

So I tried to do the same thing, but vice versa. I created a folder on the console side, updated the folder's ownership and afterwards checked if the folder appeared in the web interface, with some surprising results:

What the heck?!

Where did my folder go??!

Okay.. let me explain what's going on here. The ownCloud system basically expects two things from a file/folder in order to be indexed as a file:

• The file/folder needs to be physically available;
• The file/folder needs to exist in the ownCloud database.

Moving a file/folder into ownCloud's data directory doesn't add it to the database automatically. Luckily, the ownCloud developers were kind enough to create a command line tool called occ that can manually add missing files and folders to the ownCloud database. You can run the following snippet on your server to check where this utility is located on the local file system:

find / -name "occ" 2>/dev/null

Which outputs the following on my demo server:

All you need to do is cd into that folder (in my case /var/www/owncloud) and run the following snippet:

sudo -u www-data php occ files:scan --all

Where www-data is the user your web server runs on.

    Another option would be a command that scans a specific folder or user.  In this example, the username is "bart" and the folder to scan is  in the "files" directory called "temp2" 

 

NOTE: the path is not the full path in the file system but must start with the username.  For example my installation & data directory is located in /var/www/owncloud/data   all the user data is located in a folders (named by username), followed by a "files" folder and then any folders/directories created there after for the given user.

If you try to use a fully qualified path afer --path directive,  you will encounter the error

Unknown user 1 var

sudo -u www-data php occ files:scan --path bart/files/temp2

Further more, if you wanted to scan all the files and directories for a given user, you could run the following, where "bart" is the ownCloud username.

sudo -u www-data php occ files:scan bart

 And now, it is time for the results:

YEAH!! So this command has definitely helped. Now you can add all your files directly into your ownCloud data folder without having to sync from the desktop client.