Watch a folder for changes and perform an action

[statmonkey]

This is kind of a script/How To but think it fits best here.  Mods can move it if I have misplaced it.  It's another one of my adventures into the obscure.  I have to confess that this looks a little intimidating, it is really not.  I have just tried to be detailed and precise to make it easier.

Recently I have played around with Incron "http://inotify.aiken.cz/?section=incron&page=about&lang=en" the reason being that I was looking for a way to automatically watch a file or folder and take some action.  What I wanted to watch was the folder that my music was in and then run a command or a script to re-index my play list as kind of proof of concept. The problem I ran into was that Incron is not able to handle recursive watching, it can only watch a file or a folder not a tree.  This will explain how I solved my problem and I think opens up a fair amount of possibilities.

A couple of caveats.  I am not a python coder, I am barely a bash scriptwriter.  I have borrowed liberally and tried to credit where I got the ideas and if anyone sees a better way to do this I would love to hear it.  One problem I have not solved is watching multiple folders to run different jobs.  I have a couple of ideas of how to go about it but haven't got that far.  For now this offers a solution to my problem and perhaps someone will find it useful.

Why Would you do this?

Linux offers lots of ways to manage syncing folders and running programs (like cron) automatically but all use time as the parameter and that is inefficient for folders that change seldom and that you only want something to happen when an event takes place.  Inotify and Incron have potential but limitations and this method really has none.  If, you wanted to have your git files uploaded and committed every time you changed something in the git tree this concept would work.  In fact you could also do this for an upload if you locally develop websites for example and want to automatically upload your files when a change happened and not at a certain time every day or week.

It is fairly simple, powerful and straightforward.  For any of the following "events" you can do an action.  That action is defined as: anything from a command to a full-blown script or opening an application.  If you can conceive it in the framework it can be done.

supported events from the original README:

•    'access' - File was accessed (read) (*)
      'attribute_change' - Metadata changed (permissions, timestamps, extended attributes, etc.) (*)
     'write_close' - File opened for writing was closed (*)
     'nowrite_close' - File not opened for writing was closed (*)
     'create' - File/directory created in watched directory (*)
     'delete' - File/directory deleted from watched directory (*)
     'self_delete' - Watched file/directory was itself deleted
     'modify' - File was modified (*)
     'self_move' - Watched file/directory was itself moved
     'move_from' - File moved out of watched directory (*)
     'move_to' - File moved into watched directory (*)
     'open' - File was opened (*)
     'all' - Any of the above events are fired
     'move' - A combination of 'move_from' and 'move_to'
     'close' - A combination of 'write_close' and 'nowrite_close'

How Technical?

I find python a bit enigmatic but, as far as any real technical knowledge for anyone doing these steps it is not much.  The heavy lifting is really done by a python script and the configuration file is pretty basic. I have tried to cut any excess configuration to keep it simple.  All you really need is python installed and a couple of helpers, a terminal and a basic understanding of file structure, some command line skills and a text editor.

What is not covered?

I won't go into detail about how the python script actually works or the daemon initiator. If you are interested in any of the background information it's best if you read the sources of where I got the templates for this project.  These are as follows:

• The script watcher.sh comes from Stephen C Phillips -- http://blog.scphillips.com/2013/07/getting-a-python-script-to-run-in-the-background-as-a-service-on-boot/ and was modified to fit the needs of this tool set.
  
  The python script watcher.py originated with Andreas Gohr http://www.splitbrain.org/blog/2011-01/07-watcher_a_recursive_incron_alternative and was modified extensively to work with watcher.sh.
  
  .watcher.ini also came from Andreas Gohr above and was only slightly changed. He also provided the basis for the README to which I added extensive documentation.
  

There are some other places I picked up ideas here and there but unfortunately I have become a bit muddled on what I learned to do this and what I already knew.  Apologies to anyone who is not credited.

What do I need?

You will need python 2.7 or higher, and the pyinotify library.  The pyinotify library is just a python event notifier that works off inotify.

In Ubuntu (and Debian) you can install these with:

Code Select Expand

sudo apt-get install python python-pyinotify

For Vsido users these are probably already installed.  It is also possible you will need python-argparser but again I think most distro's have argpaser. If not, python-argparser is in the Debian repo's and you can apt-get it.

What Are the Steps

Download the package

Read the README

Place the files in the correct places/Create structure

Edit the ini file

Make sure you have the proper permissions

Initiate the daemon

Test your event and reaction

Autostart the daemon


Assumptions in this how to?

We will assume that you have a folder that holds your music and that the structure is:

Code Select Expand

/Music/Artist/Album

Or something similarly hierarchical that has several layers.  We will also assume that you have a playlists folder in ...

Code Select Expand

~/.mpd/playlists

and that you have a find command or some script you want to run.  For our purposes I am going to use a detailed command to show the power of our tool but you could use any command or script.

Steps

Download the package

Download the package either here as a tar or from the project page:
https://sourceforge.net/p/watchmonkey/code/ci/master/tree/
https://sourceforge.net/projects/watchmonkey/files/

Read the README

Ok, this isn't realistic but understand that everything that is here in this how-to is pretty much in the README.  I would like to apologize for the name of the scripts by the way but any name I could think of was already taken. Forgive me.

Set up your structure

Move to wherever you downloaded the files in a terminal.  We will need to create a folder and copy some files around. First we need somewhere to put our python script.  So create a folder called watcher in /usr/local/bin (as root for all the following):

Code Select Expand

mkdir /usr/local/bin/watcher

Next copy into the watcher folder two files.

Code Select Expand

cp watchmonkey.py README /usr/local/bin/watcher/

Make sure that watchmonkey.py is executable

Code Select Expand

chmod u+x /usr/local/bin/watcher/watchmonkey.py

Next copy over the daemon file watchmonkey.sh to /etc/init.d/

Code Select Expand

cp watchmonkey.sh /etc/init.d/watchmonkey.sh

Make sure that watchmonkey.sh is executable

Code Select Expand

chmod u+x /etc/init.d/watchmonkey.sh

Set up and edit the configuration file

Now we need to work with our configuration file .watchmonkey.ini. Copy it to the $USER directory as the user (exit root in the terminal)

Code Select Expand

cp .watch.ini ~/

Now we open that file in our favorite text editor.

Code Select Expand

geany ~/.watchmonkey.ini

There are two lines you have to change.  They are lines 25 and 67.  They are simple as follows:

• Line 25 is the path to your music folder in my case I want to watch my Jazz folder for changes.  So I enter here:

Code Select Expand

watch=/mnt/Jukebox/Music/Jazz

You would replace /mnt/Jukebox/Music/Jazz with whatever path you want to watch

• 
  Line 67 is the command you want to run.  It really can be anything for example:

Code Select Expand

echo "something changed" >  ~/mywatchedfolder.txt

for our example mine is going to thoroughly go through and make a playlist and I enter:

Code Select Expand

command=find /mnt/Jukebox/Music/Jazz -type f -name "*.mp3" -or -name "*.ogg" -or -name "*.flac" -or -name "*.m4a" > /home/stat/.mpd/playlists/JazzIndex.m3u

You would of course want to change the folder you are watching and the playlist name.  Again this is just a proof of concept.

There are some other lines you might want to check out:

The next is line 48 that sets what you want to watch. The parameters are described in lines 27-43 and can just be separated by comma's. The default is create,delete.
Line 52 allows you to set what folders in the tree or files to exclude and the same rules apply.
Line 55 allows you to set whether you want the folder watched recursively or not. Default is true
Line 58 will offer you the option of autoadd new subdirectories. Default is true
For our test we can leave these alone.

Check permissions

Check your permissions one more time and make sure the files are in the proper places:

Code Select Expand

ls -als /usr/local/bin/watcher/watchmonkey.py #should be executable

Code Select Expand

ls -als /etc/init.d/watchmonkey.sh should be executable

Code Select Expand

ls -als ~/.watcher.ini should be owned by user

Initiate the daemon

Initially we will just start the daemon from the command line and test it as a daemon. (as root)

Code Select Expand

/etc/init.d/watchmonkey.sh start

Code Select Expand

/etc/init.d/watchmonkey.sh status

This should return something like:

Starting watchmonkey.sh (via systemctl): watchmonkey.service.

and respectively:

watchmonkey.service - LSB: watch files or folder as defined
Loaded: loaded (/etc/init.d/watchmonkey.sh)
Active: active (exited) since Sat 2013-12-21 02:17:55 CST; 6s ago
Process: 2085 ExecStart=/etc/init.d/watchmonkey.sh start (code=exited, status=0/SUCCESS)
Dec 21 02:17:55 kurtvsido systemd: Starting LSB: watch files or folder as defined...
Dec 21 02:17:55 kurtvsido systemd: Started LSB: watch files or folder as defined.

So far so good (I hope)

Test your event and action
If you have followed my example testing is pretty simple.  Lets just go to our music directory.  In my case thats /mnt/Jukebox/Music/Jazz and add an mp3 file. It doesn't really have to be one but just have the watcher think it is one.  So ...

Code Select Expand

touch /mnt/Jukebox/Music/Jazz/watchmonkey.mp3

we can test this a couple of ways first:

Code Select Expand

tail -5 /var/log/syslog | tac

Should show you something like this:

Dec 21 02:17:55 kurtvsido systemd: Started LSB: watch files or folder as defined.
Dec 21 02:17:55 kurtvsido watchmonkey.sh: Starting system watchmonkey daemon:.
Dec 21 02:17:55 kurtvsido systemd: Starting LSB: watch files or folder as defined...
...

or

Code Select Expand

grep "watchmonkey.mp3" ~/.mpd/playlists/JazzIndex.m3u

Should return something like:

/mnt/Jukebox/Music/Jazz/watchmonkey.mp3

Autostarting the daemon

If you like what you see you can autostart the daemon with

Code Select Expand

sudo update-rc.d watchmonkey.sh defaults enable

Switching enable with disable will remove it from autostart.

Also remember that you should not edit the config file .watchmonkey.ini while the daemon is running. Always run:

Code Select Expand

sudo /etc/init.d/watchmonkey.sh stop

Then change the ini and restart it with the start command.  Reload is not very reliable.  On systemd setups see the README

Summary

I realized a couple of things once I got this working. First that to monitor more than one folder or file I would need separate daemons.  I am researching this but if anyone has any ideas or better python knowledge let me know.  Since jobs are defined in the ini file I should be able to code it to look at whether there are more than one job in the python script.  That is a big TODO.  The second thing is that this (to me) is really useful.  I actually have changed my initial usage to using it for git and auto committing, etc. using a script.  I have it working to check my git folders. Run a commit generically and then upload the files automatically all from a simple bash script using gpg.  I am in hopes that someone finds it useful and at the least interesting.  I have a feeling that I can find something better for remote work but am still researching that one.

Happy watchmonkeying!

[jedi]

Wow statmonkey!  That looks like it was a lot of work.  Very well done, and once again, a great contribution to the VSIDO community...

[statmonkey]

Thanks Jedi.  It wasn't really, the write up took longer than the actual coding.  It's kind of a hard thing to explain but it works great and it really is easy to implement.  I think it was worth the effort, much easier to keep folders up to date etc. If I can get it to do multiple watches in one process it will really be worth it.