Git Notes Line Endings

It is remarkable that in the year 2019 we are still dealing with an issue introduced by the workings of a manual typewriter! But alas, here we are. Windows text editors will often add CRLF (Carriage Return + Line Feed) to end of a line in a text file. Depending on your system, this may cause errors when running FDS. When viewed by a the vim text editor with the -b option, you will see a "^M" at the end of the line.

We try to automatically scrub the repo of these endings using the .gitattributes file as discussed here. However, inevitably files slip through the cracks, or we have no control over how a Submodule repo is managed. So, these notes are a reminder for how to deal with this problem and make periodic corrections.

The easiest way I have found to find and correct this problem in bulk is the following. First, at the top level (directory) of the repo run the following bash shell command, which will give you a list of files with CRLF or ^M line endings.

for i in $(grep -IURl --color --exclude="*.pdf" --exclude-dir=".git" "^M"); do echo $i; done

Note that the ^M must be entered via ctrl-v ctrl-m, otherwise the command will not work as intended. At this point you have not made any changes to the files; you have only listed them. Depending on the specific repo and file types you need to consider, you may need to amend the --exclude commands. You can add patterns like this, for example, --exclude="{*.pdf,*.exe}".

Once you have a list of files, double check that the command worked correctly. Open one of the files with vim and make sure you see ^M at the end of the lines.

$ vi -b filename

If so, manually remove the ^M with the following sed command to make sure it works on your system.

$ sed -i "s/^M//g" filename

Again ^M should be input via ctrl-v ctrl-m. Double check your file to make sure the ^M are now gone. If so, you can move on to a bulk removal as follows. Basically, you are just going to add your sed command into the bash do loop, like so:

for i in $(grep -IURl --color --exclude="*.pdf" --exclude-dir=".git" "^M"); do echo $i; sed -i "s/^M//g" $i; done

I emphasize again, make sure both ^M are input via ctrl-v ctrl-m.