Use inotifywait and rsync to automatically push code to a remote server without git (Tips for usage with PyCharm included)

I have written a little helper script that I use whenever I want to write code locally but run it remotely. This is for example useful when I cannot run the code locally because it needs one or more GPUs or is very computationally intensive.

One possibility would be to use git and push/pull each change manually. But this would obviously be too much effort for little changes (like typo fixes). Another alternative is to manually run rsync after each change. But as I am lazy, I want to run rsync automatically whenever any file in my project changes.

This is how I do it (works on Ubuntu 16.04):

while inotifywait -r -e modify,create,delete --exclude ".*(\.git|\.idea)" my-project; do
  rsync -avzh --delete my-project simon@my.remote.server:projects/
  notify-send "Sync finished" "$(date)" -i info
done

Explanation per line:

  1. blocks until inotifywait returns. This happens if any file within the my-project folder (-r means recursive) is modified, created or deleted (see -e flag). With –exclude I avoid getting events for files my IDE PyCharm creates (.git/* and .idea/*).
  2. Push all changes to the server.
    -a preserves permissions, symbolic links, etc.
    -v prints all pushed files to stdout (so you can see what was actually changed)
    -z compresses changes to speed things up and save bandwidth
    -h prints human-readable numbers
    Since I sometimes delete files, I want to reflect this also on the remote server. This is achieved by the –delete flag.
  3. Tell my Ubuntu to show me a notification as soon as the changes were sent to the server, so that I can run my code there
  4. When finished, inotifywait waits for new changes

Using this script I do not have to worry about getting my code onto the server anymore. Another advantage is that I do not have to mount the remote file system and open the mounted directory in my IDE which would be slow.

Note: The version above only ignores changes in .git and .idea folders when they are the only changes. In every other case the event is fired and rsync runs. rsync does not ignore any files, this means that .git and .idea will be pushed to the server. The effect is that whenever you change branches locally or change project settings, the changes will also be sent to the server. If you do not want this, add this to the rsync parameters:

-exclude “.idea” –exclude “.git”

It always depends on your use case if it is useful to synchronize the git repository or not.

Important for PyCharm users

It took me a while to figure out why my script showed me strange events when I was editing code. I saw messages like:

my-dir/ CREATE main.py___jb_tmp___
my-dir/ MODIFY main.py___jb_tmp___
my-dir/ DELETE main.py___jb_old___
my-dir/.idea/ CREATE workspace.xml___jb_tmp___
my-dir/.idea/ MODIFY workspace.xml___jb_tmp___
my-dir/.idea/ DELETE workspace.xml___jb_old___
my-dir/.git/ CREATE index.lock
my-dir/.git/ DELETE index.lock

But I saw no event for the real main.py file. After researching a bit I found that PyCharm has a “safe write” mode which was enabled. This mode has the effect that PyCharm always writes changes to a temporary file first and if this is successful, it deletes the actual file and moves the temporary file to its position. You can disable the “safe write” mode in the settings: Appearance & Behaviour -> System Settings -> Synchronisation -> remove tick at “Use safe write […]”.

I hope this post was helpful for you. If you have useful additions, please use the comment box.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.