feature: added support for MSVC compiler. #1222

geniuss99 · 2018-01-07T12:01:34Z

I hereby granted the copyright of the changes in this pull request to the authors of this lua-nginx-module project.

These patches will allow lua-nginx-module to be compiled using MSVC compiler. In order to make successful builds nginx files MUST also be patched. Overall there are two patches: one for nginx, another one - for lua-nginx-module (please, see an archive in attachment). The explanations for the patches are given below.

Patch for Nginx.

The following measures are provided:

Disable use of precompiled headers. This is because precompiled header which is produced for <ngx_config.h> file is used for every ".c" file with every "cl.exe" command (due to the nature of Makefile generated by nginx configure scripts). But if we do it like this then (according to MSVC rules) "#include <ngx_config.h>" line MUST be the first line for every ".c" file compiled. This is true for nginx sources but not for nginx-lua-module and ngx_devel_kit-0.3.0. So either the code for these modules should be rewritten or separate Makefile for them must be created. I decided that switching off the use of precompiled headers is the best option for MSVC.
Update "auto/feature" helper script to support MSVC compiler syntax. By default only gcc-style test command is supported. They are not compatible.
Update and create new "auto/lib/openssl/*" scripts to support MSVC x64 compiler. Also for x32 builds NASM is used instead of no-asm variant. It looks like this variant is faster than building without asm.
Please, see the following link: http://openssl.6102.n7.nabble.com/asm-vs-no-asm-performance-test-results-and-security-concern-td36571.html

I suppose patch for nginx (see attach) should go here (it's not for me to decide though):
https://github.com/openresty/openresty/tree/master/patches.

Patch for nginx-lua-module

The following changes are made:

Config file is split into two parts: for gcc (or clang) variant and msvc one. As far as I understand gcc and clang are the only compilers currently supported.
Module sources (common code for both configs) are moved to a separate file.

How to compile using MSVC

Prerequisites

To build nginx on the Microsoft Win32 platform you need:

MS Visual C compiler 14.0 or higher (earlier version might also work but I didn't test it).
Msys;
Perl (for OpenSSL); 
NASM 2.13.02 or higher (for OpenSSL);
LuaJIT 2.0.1-beta3 libraries compiled with MSVC (must be done separately);
Nginx 1.13.7;
OpenSSL 1.0.2m;
PCRE 8.41;
ZLib 1.2.11;
lua-nginx-module 0.10.12.rc1 (patched version).

STEP 1

Set MSVC environment (x86 or x64):
"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" x86
or
"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" x64

STEP 2

Add path for MSYS and NASM into %PATH% environment variable:
SET PATH=%PATH%;J:\Tools\compilers\nasm-2.13.02;C:\MinGW\msys\1.0\bin

STEP 3

Start MSYS bash:
bash

STEP 4

Download nginx sources from the hg.nginx.org repository and patch them with nginx-1.13.7.1-lua-nginx-msvc-support.patch:

tar -xzf nginx-1.13.7.tar.gz
cd %NGINX_ROOT_DIR%
patch.exe -p1 < nginx-1.13.7.1-lua-nginx-msvc-support.patch

STEP 5

Create a build and lib directories and unpack zlib, PCRE, OpenSSL libraries sources into lib directory:

cd %NGINX_ROOT_DIR%
mkdir objs
mkdir objs/lib
cd objs/lib
tar -xzf ../../pcre-8.41.tar.gz
tar -xzf ../../zlib-1.2.11.tar.gz
tar -xzf ../../openssl-1.0.2m.tar.gz

STEP 6

Put already compiled LuaJIT library into the lib folder:
tar -xzf ../../luajit-2.0.1-msvc.tar.gz

STEP 7

Create "modules" directory and unpack lua-nginx-module-0.10.12rc1 (patched version) and ngx_devel_kit-0.3.0 modules there:

cd %NGINX_ROOT_DIR%
mkdir objs/modules
tar -xzf ../../lua-nginx-module-0.10.12rc1.tar.gz
tar -xzf ../../ngx_devel_kit-0.3.0.tar.gz

STEP 8

Define LuaJIT variables used by lua-nginx-module:

export LUAJIT_INC='%NGINX_ROOT_DIR%/objs/lib/LuaJIT/include'
export LUAJIT_LIB='%NGINX_ROOT_DIR%/objs/lib/LuaJIT/'

STEP 9

Configure nginx:

cd %NGINX_ROOT_DIR%
configure \
    --with-cc=cl \
    --with-cc-opt="-DFD_SETSIZE=10240" \
    --builddir=objs \
    --build="Custom Build" \
    --prefix= \
    --conf-path=conf/nginx.conf \
    --pid-path=logs/nginx.pid \
    --lock-path=temp/nginx.lock \
    --http-log-path=logs/access.log \
    --error-log-path=logs/error.log \
    --sbin-path=nginx.exe \
    --http-client-body-temp-path=temp/client_body_temp \
    --http-proxy-temp-path=temp/proxy_temp \
    --http-fastcgi-temp-path=temp/fastcgi_temp \
    --http-scgi-temp-path=temp/scgi_temp \
    --http-uwsgi-temp-path=temp/uwsgi_temp \
    --with-select_module \
    --with-http_ssl_module \
    --with-http_v2_module \
    --with-pcre=objs/lib/pcre-8.41 \
    --with-pcre-jit \
    --with-zlib=objs/lib/zlib-1.2.11 \
    --with-openssl=objs/lib/openssl-1.0.2m \
    --add-module=objs/modules/ngx_devel_kit-0.3.0 \
    --add-module=objs/modules/lua-nginx-module-0.10.12rc1

STEP 10

Compile and build:
nmake

MSVC Run-Time library

By default nginx uses so called multithread, static version of the run-time library (/MT linker switch). I didn't change this so basically LuaJIT should also be compiled in a static version (as a static library).

Another option is to use multithread, DLL-specific version of the run-time library (/MD linker switch).
This way LuaJIT must be compiled as a .dll library. For this LIBC variable in "auto/cc/msvc" file must be changed from "-MT" to "-MD" (though i didn't try this variant).

I guess dll-specific version of RTL is more preferable for a general use but it's for OpenResty developers to decide. Anyway these patches are a good starting point for MSVC support.

Thank you for attention and sorry if I made a lot of text :)

[Update] For additional info about how MSVC CRT's work please see:
https://msdn.microsoft.com/en-us/library/abx4dbyh.aspx

agentzh · 2018-01-07T20:14:53Z

@geniuss99 Thanks for your hard work on this! It seems that the official nginx can already build on Win32 without patching:

http://nginx.org/en/docs/howto_build_on_win32.html

Maybe we should instead adjust the ngx_lua, ngx_devel_kit, and etc modules to accommodate nginx's official Win32 way of doing things?

Basically we try hard to minimize OpenResty's own patches for the nginx core unless absolutely necessary. Or is there anything significant I'm overlooking here?

geniuss99 · 2018-01-08T08:46:31Z

It seems that the official nginx can already build on Win32 without patching:

Yes, it can. Pure nginx (without lua-nginx-module) with OpenSSL, Zlib and PCRE can be built without any problems. Though you won't be able to build x64 version with OpenSSL because nginx configure scripts support only x32 version of OpenSSL for msvc (see %NGINX_ROOT_DIR%/auto/lib/openssl/makefile.msvc, line 9, "perl Configure VC-WIN32 no-shared").

Maybe we should instead adjust the ngx_lua, ngx_devel_kit, and etc modules to accommodate nginx's official Win32 way of doing things?

The way I described is actually official one but more detailed and with some minor changes :)
As for adapting the ngx_lua and ngx_devel_kit modules without nginx patching technically it's possible. Though you won't be able to produce x64 builds (again, because of OpenSSL issue I described earlier). To do this we need to solve 2 problems.

Problem 1: use of precompiled header <ngx_config.h>

I described the problem in details in my first post.
Solutions:

Add #include <ngx_config.h> code into every ".c" file in all current and future modules (ngx_lua and ngx_devel_kit for now). Looking on the code of ngx_lua it's possible because every ".c" file includes "ddebug.h" which in turn includes "ngx_config.h". Same goes for ngx_devel_kit. So solution 1 is viable. We just need to patch every ".c" file :)
Write separate Makefile with custom compiler flags for ngx_lua and ngx_devel_kit modules. Run it by nginx's global Makefile (this solution is used by nginx with OpenSSL, Zlib and PCRE libraries).

Problem 2: ngx_lua's config script

Current version of ngx_lua's config script uses %NGINX_ROOT_DIR%/auto/feature helper script to check for features. This helper script generates ".c" test file and runs gcc-style compiler to build it.
Solution: write your own shell code to test for features, msvc compatible.
Can it be done? Yes, it can.

That's it. Once those 2 problems are solved ngx_lua can be compiled with stock nginx (x32 builds only).

As for my nginx patches I think it's possible to submit them to nginx developers (they don't break anything actually, except for disabling of precompiled headers which can be circumvented through new configure option).

agentzh · 2018-01-08T19:15:30Z

@geniuss99 Sorry, I didn't know nginx only supports 32-bit OpenSSL on windows. That's indeed unfortunate and I can now see the justification of patching the nginx core. Thank you for your detailed explanation!

I had a quick glance at your ngx_lua patch and it looks fine to me, though we'll definitely have a closer look before it can be merged.

Regarding to the nginx core patch, will you please create a pull request to the openresty/openresty repository so that it's easier for us to review and merge. OpenResty's own nginx patches all go here:

https://github.com/openresty/openresty/tree/master/patches

We're very interested in shipping official Win32 and Win64 binaries produced by the MSVC toolchain. We currently only ship a slow MinGW binary build for Wn32.

Thank you again for the hard work on this!

agentzh · 2018-01-08T19:26:27Z

@geniuss99 If I understand it correctly, the other 3rd-party nginx C modules in OpenResty which do not involve feature tests do not require a separate config.msvc file, right?

agentzh · 2018-01-08T19:29:21Z

config

+        echo "using untested compiler to build the module";
+        . $ngx_addon_dir/auto/config.gcc;
+    ;;
+esac


We need to add back the following commented lines at the end of this file. They are meaningful to OpenResty's build toolchain:

#CFLAGS=$"$CFLAGS -DLUA_DEFAULT_PATH='\"/usr/local/openresty/lualib/?.lua\"'" #CFLAGS=$"$CFLAGS -DLUA_DEFAULT_CPATH='\"/usr/local/openresty/lualib/?.so\"'"

agentzh · 2018-01-08T19:33:10Z

@geniuss99 Have you tried loading lua-cjson and ffi.load("LIBRARY") in your Win32/Win64 binary build? Do they work as expected? And how about lua-resty-core?

geniuss99 · 2018-01-09T08:17:12Z

@geniuss99 Sorry, I didn't know nginx only supports 32-bit OpenSSL on windows.

It's not that nginx itself doesn't support x64 OpenSSL - its code base is fully compatible with x64 OpenSSL.
But its config scripts used to build it don't support it. This problem exists because Nginx developers don't pay much attention to Windows platform so I guess they haven't updated their MSVC config scripts for quite a long time. My patch only fixes this issue, it doesn't change any sources of nginx (".c" or ".h" files).

@geniuss99 If I understand it correctly, the other 3rd-party nginx C modules in OpenResty which do not involve feature tests do not require a separate config.msvc file, right?

Theoretically, yes. If they don't have other incompatibilities. I haven't checked all of them.

@geniuss99 Have you tried loading lua-cjson and ffi.load("LIBRARY") in your Win32/Win64 binary build? Do they work as expected? And how about lua-resty-core?

Oh I'm not sure of that :) To tell you the truth I only use nginx +lua-nginx-modulein the project I'm currently working on (for now at least). I don't use the whole OpenResty application.

Basically to make all OpenResty MSVC-compatible we need to assure the following:

All ".c" files of all modules can actually be compiled with MSVC compiler.
Build scripts should generate correct Makefile for MSVC.

I had a quick glance at OpenResty sources. There are many! modules in the folder "bundles". Some of them are pure Lua modules (they pose no problem), some - C modules. And some of these C modules use incompatible config scripts which need to be patched (i.e. ngx_stream_lua-0.0.3, drizzle-nginx-module-0.1.10). Also there is also a big configure script written in Perl. It should also be updated to support MSVC. I guess it's not an quick task :) Though nothing impossible.

@agentzh What modules are of utmost importance to OpenResty application in your opinion? I can examine them and try to compile with MSVC.

agentzh · 2018-01-09T22:49:14Z

@geniuss99 Thank you for your help here! The other most important ones are ngx_stream_lua and lua-resty-core, and lua-cjson. I guess ngx_stream_lua should be VERY similar to ngx_http_lua (i.e., this repository). It would be great if you can help supporting these on native Windows builds (Win32 & Win64).

BTW, ngx_lua will soon rely on lua-resty-core to work properly soon. The existing CFunction-based Lua API is slow and can also be buggy in some corner cases, and should be replaced by the FFI-based implementation in lua-resty-core at all times.

geniuss99 · 2018-01-10T13:22:51Z

okay, I'll look into the modules you've mentioned.

agentzh · 2018-01-10T20:41:07Z

@geniuss99 Great. Thank you very much!

geniuss99 · 2018-01-21T19:49:56Z

I've got some news. Many problems are now solved.

I took OpenResty's forks of Nginx and LuaJIT. Looks like both are pretty heavily patched compared to original versions (not sure if vanilla LuaJIT might still be used for OpenResty).

I've successfully made a custom build of OpenResty (stripped version) using MSVC compiler with the following packets: Nginx 1.13.6 (OpenResty's fork), LuaJIT 2.1.0 (OpenResty's fork), lua-nginx, stream-nginx, lua-cjson.

Each of them needs patching for the whole product to compile and work :)

Nginx 1.13.6
Needs some non-essential patching of sources and build scripts. Build scripts are updated comparing to my previous patches. Main change - I've switched from "/MT" to "/MD" compile option. This changes the CRT used (see my first post for details). This is more correct for such product as OpenResty which depends on external dlls.

LuaJIT 2.1.0
Needs some non-essential patching of sources.

lua-nginx
New build scripts for MSVC only. Codebase is untouched.

stream-nginx
Needs some non-essential patching of sources and new build scripts.

lua-cjson
Needs some non-essential patching of sources. Also I wrote a separate build script for MSVC.
Compiles with many warnings, mainly because of incorrect conversions from big to small data types.

lua-resty-core
It's just a bunch of lua files which depend on *_ffi_* functions inside lua-nginx and stream-nginx modules. I just copy these files into the "lua" folder near the nginx.exe file. Nothing to port here actually.

So in the end I've recreated the same files/folders structure as in official OpenResty package compiled using MinGW GCC. Now, the most interesting question is: does it all work?

@agentzh Have you tried loading lua-cjson

Yes, loading works. Functions from module work.

@agentzh ffi.load("LIBRARY")

Yes, ffi mechanism works.

@agentzh And how about lua-resty-core?

I've tested some of the functions and they work without problems.

@agentzh ngx_stream_lua

The example from official documentation works as expected.

I've used the following config to test the modules:

worker_processes 1;
events {
    worker_connections 10240;
}

stream {
    log_format LOG2 '$time_local	$remote_addr	$status	$upstream_addr';

    # define a TCP server listening on the port 1234:
    server {
        listen 1234;

        content_by_lua_block {
            ngx.say("Hello, Lua!")
        }
    }
    
    server {
        listen 4343 ssl;
        
        access_log logs/access-stream.log LOG2;
        error_log  logs/error-stream.log info;
       
        
        ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
        #ssl_ciphers         AES128-SHA:AES256-SHA:RC4-SHA:DES-CBC3-SHA:RC4-MD5;
        ssl_certificate     certs/server_bundled_ec.crt;
        ssl_certificate_key certs/server_private_key_ec.pem;
        ssl_session_cache   shared:SSL:10m;
        ssl_session_timeout 10m;

        content_by_lua_block {
            local sock = assert(ngx.req.socket(true))
            local data = sock:receive()  -- read a line from downstream
            if data == "thunder!" then
                ngx.say("flash!")  -- output data
            else
                ngx.say("boom!")
            end
            ngx.say("the end...")
        }
    }    
}

http {
    server_tokens off;

    resolver 8.8.8.8 8.8.4.4 ipv6=off;
    resolver_timeout 5s;

    recursive_error_pages on;
    log_format LOG1 '$time_local	$remote_addr	$host	$request	$status	$body_bytes_sent	$request_time	$upstream_addr	$upstream_status	$upstream_http_server	$http_referer	$args	$http_user_agent	$http_accept_encoding';

    server {
        listen      192.168.1.1:80  default_server;
        listen      192.168.1.1:443 default_server ssl;
        
        server_name _;

        access_log logs/access-http.log LOG1;
        error_log  logs/error-http.log info;

        charset utf-8;
        
        # RSA Cert
        ssl_certificate      certs/server_bundled_rsa.crt;
        ssl_certificate_key  certs/server_private_key_rsa.pem;

        # EC Cert
        ssl_certificate      certs/server_bundled_ec.crt;
        ssl_certificate_key  certs/server_private_key_ec.pem;

        ssl_ciphers   HIGH:!aNULL:!MD5;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers   off;
        
      
        location / {
            content_by_lua_block {
                -- CJson test
                local cjson = require "cjson"
                local json = cjson.encode({
                    foo = "bar",
                    some_object = {},
                    some_array = cjson.empty_array
                })
                
                ngx.say(json)
                
                -- resty-core tests
                require "resty.core"
                local md5Hash = ngx.md5(json)
                local time = ngx.now()
                local pid = ngx.worker.pid()
                local workerCount = ngx.worker.count()
                
                ngx.say(md5Hash)
                ngx.say(time)
                ngx.say(pid)
                ngx.say(workerCount)
                
                ngx.eof()
                                
                -- LuaJIT FFI test
                local ffi = require("ffi")

                ffi.cdef[[
                    typedef struct {int unused;} *HWND;
                    int __stdcall MessageBoxA(HWND hWnd, const char* lpText, const char* lpCaption, unsigned int uType);
                ]]
                
                local user32 = ffi.load("User32.dll")
                user32.MessageBoxA(nil, "Nginx LuaJit FFI Window", "This is a MessageBox from Nginx :)", 65)
            }
            
            header_filter_by_lua_block {
                ngx.log(ngx.ERR, "!!!header_filter_by_lua_block!!!")
            }
        }
    }
}

There are two remaining issues as of now:

1. Support for perl's test scripts in "\t" folder.

I tried to port perl's Test::Nginx module to Win32 platform (without Msys subsystem) but it's not easy. Too many issues, heavily tied to POSIX-compliant platform.

2. External build script.

There needs to be some external build script which automates build steps which I do manually now (there are not so many of them actually). It can be separate script or your big perl script might be updated. I would prefer to write a separate script for MSVC though.

@agentzh I've attached an archive with all the patches. Please, examine them and give your opinion.
If you need more details don't hesitate to ask. Thanks.

OpenResty-MSVC-Patches.tar.gz

agentzh · 2018-01-21T22:01:38Z

@geniuss99 This is great progress! We'll try your build and patches out ourselves as soon as we can manage. Many thanks!

agentzh · 2018-01-21T22:03:20Z

@geniuss99 Right. Test::Nginx is not easy to get ported over to Win32. I knew that :)

chipitsine · 2018-06-16T11:11:20Z

do you mind of getting this build on app veyor ?

mergify · 2020-06-26T00:39:39Z