Null object exception

[HurryStarfish]

I downloaded SCT and tried to use it on the tests that come with NG.

I changed the bottom part of sct.bmx to

'adjust compiler path for this test class
TTestCompiler.baseConfig.Add("bmk_path", "C:\Programmierung\BlitzMax-NG\bin\bmk.exe")

'adjust base config for all instances of that type
TTestCompiler.baseConfig.Add("app_type", "console")
TTestCompiler.baseConfig.Add("app_arch", "x86") 'unused yet
TTestCompiler.baseConfig.Add("debug", "0")
TTestCompiler.baseConfig.Add("threaded", "0")
TTestCompiler.baseConfig.Add("deleteBinaries", "1") 'delete binaries afterwards
TTestCompiler.baseConfig.Add("make_mods", "0") 
TTestCompiler.baseConfig.Add("quick", "0") 
TTestCompiler.baseConfig.fileUri = "baseConfig"

Global testManager:TTestManager = New TTestManager
testManager.AddTestsFromDirectory("C:\Programmierung\BlitzMax-NG\src\bcc\tests")
testManager.RunTests()

and ran it, but it caused a null object exception in function StandardIOAvailable. Same thing happens if I use the bmk path to the legacy BlitzMax compiler instead (I compiled SCT in legacy as well).
Am I doing something wrong here?

[GWRon]

Will try this evening. What OS do you run? Am not sure if windows10 limits accesses between "different" basepaths.

[HurryStarfish]

I use Windows 7.
Access permissions shouldn't be the issue; it's running several tests successfully before crashing:

Building sct
Compiling:sct.bmx
flat assembler version 1.68 (1048575 kilobytes memory)
3 passes, 3460 bytes.
Linking:sct.debug.exe
Executing:sct.debug.exe
=== STARTING TESTS ===

• AMOUNT OF TESTS: 54

• RUNNING TEST: C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/language/array_01.bmx
  COMMAND: C:\Programmierung\BlitzMax\bin\bmk.exe makeapp -a -t console -r -g x86 -o "C:/Programmierung/BlitzMax-NG/src/bcc/tests/framework/language/array_01.exe" "C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/language/array_01.bmx"
  -> OK

• RUNNING TEST: C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/language/array_slice.bmx
  COMMAND: C:\Programmierung\BlitzMax\bin\bmk.exe makeapp -a -t console -r -g x86 -o "C:/Programmierung/BlitzMax-NG/src/bcc/tests/framework/language/array_slice.exe" "C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/language/array_slice.bmx"
  -> OK

• RUNNING TEST: C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/language/arrays_0.bmx
  COMMAND: C:\Programmierung\BlitzMax\bin\bmk.exe makeapp -a -t console -r -g x86 -o "C:/Programmierung/BlitzMax-NG/src/bcc/tests/framework/language/arrays_0.exe" "C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/language/arrays_0.bmx"
  -> OK

• RUNNING TEST: C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/language/arrays_02.bmx
  COMMAND: C:\Programmierung\BlitzMax\bin\bmk.exe makeapp -a -t console -r -g x86 -o "C:/Programmierung/BlitzMax-NG/src/bcc/tests/framework/language/arrays_02.exe" "C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/language/arrays_02.bmx"
  -> OK

• RUNNING TEST: C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/language/arrays_03.bmx
  COMMAND: C:\Programmierung\BlitzMax\bin\bmk.exe makeapp -a -t console -r -g x86 -o "C:/Programmierung/BlitzMax-NG/src/bcc/tests/framework/language/arrays_03.exe" "C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/language/arrays_03.bmx"
  -> OK

• RUNNING TEST: C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/language/comparison_01.bmx
  COMMAND: C:\Programmierung\BlitzMax\bin\bmk.exe makeapp -a -t console -r -g x86 -o "C:/Programmierung/BlitzMax-NG/src/bcc/tests/framework/language/comparison_01.exe" "C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/language/comparison_01.bmx"
  -> OK

• RUNNING TEST: C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/language/dotdot_01.bmx
  COMMAND: C:\Programmierung\BlitzMax\bin\bmk.exe makeapp -a -t console -r -g x86 -o "C:/Programmierung/BlitzMax-NG/src/bcc/tests/framework/language/dotdot_01.exe" "C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/language/dotdot_01.bmx"
  -> OK

• RUNNING TEST: C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/language/end_01.bmx
  COMMAND: C:\Programmierung\BlitzMax\bin\bmk.exe makeapp -a -t console -r -g x86 -o "C:/Programmierung/BlitzMax-NG/src/bcc/tests/framework/language/end_01.exe" "C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/language/end_01.bmx"
  -> OK

• RUNNING TEST: C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/language/exception_01.bmx
  COMMAND: C:\Programmierung\BlitzMax\bin\bmk.exe makeapp -a -t console -r -g x86 -o "C:/Programmierung/BlitzMax-NG/src/bcc/tests/framework/language/exception_01.exe" "C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/language/exception_01.bmx"
  -> OK

• RUNNING TEST: C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/language/fp_arrays_01.bmx
  COMMAND: C:\Programmierung\BlitzMax\bin\bmk.exe makeapp -a -t console -r -g x86 -o "C:/Programmierung/BlitzMax-NG/src/bcc/tests/framework/language/fp_arrays_01.exe" "C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/language/fp_arrays_01.bmx"

Process terminated

[GWRon]

You could run sct in maxide to debug it there...maybe it exposes what is failing. But as said I will try on my own when home. Did never really try it on windows.

[GWRon]

I just tried it - and in the "ifthen_01.bmx" sample/test it took a long while (so 10 seconds or so) and ended with a segfault (when run as debug from within MaxIDE with the "Program - command line" menu option configured accordingly to run the tests).

I then executed it via gdb and it finished the tests:

=== FINISHED TESTS ===
* TIME: 30664ms
* FAILED: 1
* ERROR: 6
* OK: 53
[Inferior 1 (process 9500) exited normally]

Cleared the test folder, run it again from within MaxIDE ... and same results as when run from within gdb, so no "segfault".

Will try tomorrow on another machine. If you can "repeatingly" receive this test aborts, open up sct.bmx in MaxIDE, enable "debug", and then adjust "program - command line" to contain the path to the tests. "base.conf" within the test directory could be used to adjust "relative URI to bmk" from your SCT folder (so might be something like "../../../BlitzMax-NG/bin/bmk.exe")

[HurryStarfish]

Running sct from MaxIDE, with sct.bmx modified like in my initial post, I consistently get it when its trying to test fp_arrays_01.bmx.
The code in question is

	Method StandardIOAvailable:Int()
		Return standardIO.bufferpos Or standardIO.readAvail()
	End Method

with the callstack RunTests -> Run -> Validate -> Eof -> IOAvailable -> StandardIOAvailable.
standardIO is Null, hence the exception.

[GWRon]

While we could fix that crash by checking for "standardio and..." before doing the other stuff, it does not fix the underlying problem. Something makes the stream not useable/openable.

[GWRon]

Hmm, the standardIO variable is created within TCodeTesterProcess.RunProcess().

	Method RunProcess:TCodeTesterProcess()
		Local infd:Int, outfd:Int, errfd:Int

		handle = fdProcess(name, Varptr infd, Varptr outfd, Varptr errfd, flags)
		If Not handle Then Return Null

'		standardIO = TPipeStreamUTF8.Create(infd, outfd)
'		errorIO = TPipeStreamUTF8.Create(errfd, outfd)
		standardIO = TPipeStream.Create(infd, outfd)
		errorIO = TPipeStream.Create(errfd, outfd)

		processes.AddLast(Self)
		Return Self
	End Method

Maybe we run into some kind of "spawn/fork" limit or so?

Aside of that I am not sure if we properly finish created processes (maybe some of them "lock up") and so things might lead to a "concurrent spawned processes"-limit or so? Am currently checking that

[GWRon]

I committed some "cleanup + throw in case of null" lines - but this won't fix it for now.

I use "vanilla" to compile in non-threaded mode. With "NG" you automatically build the threaded variant. The threaded builds for now have issues - which I cannot explain yet.

The issue is, that the streams reading standardIO and errorIO receive data from the other tests.

* RUNNING TEST: ../../../Tools/BlitzMaxNG/bin/tests/framework/language/end_01.bmx
  COMMAND: /home/ronny/Arbeit/Programmieren/Tools/BlitzMaxNG/bin/bmk makeapp -a -t console -r -g x86 -o "../../../Tools/BlitzMaxNG/bin/tests/framework/language/end_01" "../../../Tools/BlitzMaxNG/bin/tests/framework/language/end_01.bmx"
  -> OK
[ 10%] Processing:exception_01.bmx
[ 90%] Compiling:exception_01.bmx.console.release.linux.x86.c
* RUNNING TEST: ../../../Tools/BlitzMaxNG/bin/tests/framework/language/fp_arrays_01.bmx
  COMMAND: /home/ronny/Arbeit/Programmieren/Tools/BlitzMaxNG/bin/bmk makeapp -a -t console -r -g x86 -o "../../../Tools/BlitzMaxNG/bin/tests/framework/language/fp_arrays_01" "../../../Tools/BlitzMaxNG/bin/tests/framework/language/fp_arrays_01.bmx"
  VALIDATION: FAILED
  expected:
  -----
  func1 1 1 1.0000000000000000
  func2 2 2 2.0000000000000000
  func3 3 3 3.0000000000000000
  ------
  received:
  -----
  ------
  -> FAILED
* RUNNING TEST: ../../../Tools/BlitzMaxNG/bin/tests/framework/language/function_01.bmx
  COMMAND: /home/ronny/Arbeit/Programmieren/Tools/BlitzMaxNG/bin/bmk makeapp -a -t console -r -g x86 -o "../../../Tools/BlitzMaxNG/bin/tests/framework/language/function_01" "../../../Tools/BlitzMaxNG/bin/tests/framework/language/function_01.bmx"
  -> OK
[100%] Linking:exception_01
* RUNNING TEST: ../../../Tools/BlitzMaxNG/bin/tests/framework/language/fp_local_01.bmx
  COMMAND: /home/ronny/Arbeit/Programmieren/Tools/BlitzMaxNG/bin/bmk makeapp -a -t console -r -g x86 -o "../../../Tools/BlitzMaxNG/bin/tests/framework/language/fp_local_01" "../../../Tools/BlitzMaxNG/bin/tests/framework/language/fp_local_01.bmx"
  -> OK
* RUNNING TEST: ../../../Tools/BlitzMaxNG/bin/tests/framework/language/exception_01.bmx
  COMMAND: /home/ronny/Arbeit/Programmieren/Tools/BlitzMaxNG/bin/bmk makeapp -a -t console -r -g x86 -o "../../../Tools/BlitzMaxNG/bin/tests/framework/language/exception_01" "../../../Tools/BlitzMaxNG/bin/tests/framework/language/exception_01.bmx"
  VALIDATION: FAILED
  expected:
  -----
  start TRY
  assignment to null is not allowed
  end TRY
  ------
  received:
  -----
  ------
  -> FAILED

You see multiple issues there: the received strings are most often "empty" instead of containing valid data.
Also it receives output of other threads when compiling a validation-binary. It should not do this:

* RUNNING TEST: ../../../Tools/BlitzMaxNG/bin/tests/framework/language/end_01.bmx
  COMMAND: /home/ronny/Arbeit/Programmieren/Tools/BlitzMaxNG/bin/bmk makeapp -a -t console -r -g x86 -o "../../../Tools/BlitzMaxNG/bin/tests/framework/language/end_01" "../../../Tools/BlitzMaxNG/bin/tests/framework/language/end_01.bmx"
  -> OK
[ 10%] Processing:exception_01.bmx
[ 90%] Compiling:exception_01.bmx.console.release.linux.x86.c
* RUNNING TEST: ../../../Tools/BlitzMaxNG/bin/tests/framework/language/fp_arrays_01.bmx

(as you see it talks about "exception_01.bmx" which another test. Another test then contains the [100%] of this exception_01 compilation).

I am not sure but I assume that "pub.mod/FreeProcess.mod" might not be threadsafe (PIDs of the executed commands) or something with its TPipeStream (on which the PipeStream in SCT is based on).

Nonetheless I think "threads" and your "null stream" might be two different bugs.

Also I have to note, that @woollybah got reports by me about "mixed output" already. While building something with "bmk.threaded.debug"-builds I got some stuff printed within other lines - so thread output was "mixed" into one stream. So the "[100%]" bug of above might be connected to this issue as well.

[HurryStarfish]

I use "vanilla" to compile in non-threaded mode.

I've tried it with legacy/vanilla in both threaded and non-threaded, as well as debug and release mode. The error happens regardless.

[HurryStarfish]

Hmm, the standardIO variable is created within TCodeTesterProcess.RunProcess().

Maybe we run into some kind of "spawn/fork" limit or so?

Aside of that I am not sure if we properly finish created processes (maybe some of them "lock up") and so things might lead to a "concurrent spawned processes"-limit or so? Am currently checking that

That might be the case. When I run these tests, there are a lot of calls to TCodeTesterProcess.RunProcess happening (one or two for each test), but none to TCodeTesterProcess.Close. So pipe streams are continuously being created without being cleaned up up until the crash.

[GWRon]

If a process is ending thr handle gets nulled...and the process is removed from a container list (tlist) but maybe GC is not kicking in fast enough to get rid of the streams. The new version should close streams earlier (on forked process end). Does it help?

[HurryStarfish]

I don't see a Finalizer in TCodeTesterProcess or in TPipeStream itself to clean up the unmanaged handles it allocates, so I don't think the GC could do anything here.

I just tried the new commit, and it ran to completion (with lots of failed tests and a crash in one of the tests but not from SCT itself).
However, it somehow managed to kill my explorer.exe - or rather, made one explorer window freeze and several others glitch out so badly that I had to kill it myself. Tried to take a screenshot of it, but that glitched as well and didnt even show the broken windows. That's quite impressive.
Now that I'm taking a closer look at PUB.FreeProcess, that thing seems like it is badly in need of an update for NG. It's using ints everywhere to pass around handles (which are pointers), so it's not too surprising that things might go wrong with that. That doesn't seem to explain what I just experienced however, since I just realized I was using legacy BlitzMax for this test run...

[GWRon]

Streams should get closed once the process is no longer "alive". There is some function called to remove "zombies". This also calls CloseStreams() if the process ended meanwhile ... which removes connections to the streams. Maybe you could add another "print" to the functions outputting the "processes.Count()" and also a debugprint when streams get closed. Maybe on your computer this does not happen at all?

[GWRon]

If you open up "base.processes.bmx" and replace

	Method CloseStreams()

with

	Method CloseStreams()
		print "CloseStreams() - name: " + name

You should see this message printed during execution:

./bmk makeapp -t console -quick -r -x "/home/ronny/Arbeit/Programmieren/Projekte/BlitzMaxNG.project/SimpleCodeTester-master/sct.bmx"
[ 51%] Processing:base.processes.bmx
[..]
[100%] Linking:sct
Executing:sct
=== STARTING TESTS ===
* AMOUNT OF TESTS: 65
* RUNNING TEST: ../../../Tools/BlitzMaxNG/bin/tests/framework/language/array_01.bmx
  COMMAND: /home/ronny/Arbeit/Programmieren/Tools/BlitzMaxNG/bin/bmk makeapp -a -t console -r -g x86 -o "../../../Tools/BlitzMaxNG/bin/tests/framework/language/array_01" "../../../Tools/BlitzMaxNG/bin/tests/framework/language/array_01.bmx"
CloseStreams() - name: /home/ronny/Arbeit/Programmieren/Tools/BlitzMaxNG/bin/bmk makeapp -a -t console -r -g x86 -o "../../../Tools/BlitzMaxNG/bin/tests/framework/language/array_01" "../../../Tools/BlitzMaxNG/bin/tests/framework/language/array_01.bmx"
  -> OK
* RUNNING TEST: ../../../Tools/BlitzMaxNG/bin/tests/framework/language/array_slice.bmx
[...]

If "CloseStreams()" is not called, then fdProcessStatus is still returning a value > 0 (values interpreted as "true" in BlitzMax).

You could also replace (in the same file):

	Method RunProcess:TCodeTesterProcess()

with

	Method RunProcess:TCodeTesterProcess()
		print "RunProcess: "+processes.Count()+" processes still running."

And it should always print:

RunProcess: 0 processes still running.

(at least it does so on my computer).

[HurryStarfish]

I dared run it again; this time it didn't break explorer.
Streams seem to be getting closed properly with the latest commit; RunProcess and CloseStreams calls alternate and processes.Count() at these points returns 0 and 1 respectively.

I'm seeing a lot of tests (17 out of 58) fail, however, most of them erroneously, due to their output getting cut off. As an example:

* RUNNING TEST: C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/types/hierarchy_02.bmx
  COMMAND: C:\Programmierung\BlitzMax-NG\bin\bmk.exe makeapp -a -t console -r -g x86 -o "C:/Programmierung/BlitzMax-NG/src/bcc/tests/framework/types/hierarchy_02.exe" "C:\Programmierung\BlitzMax-NG\src\bcc\tests/framework/types/hierarchy_02.bmx"
  VALIDATION: FAILED
  expected:
  -----
  Super.GetProp: A  =  A
  Self.GetProp : B =/= A
  Self.prop    : B =/= A
  Self.prop2   : A  =  A
  Self.GetGlob : A  =  A
  Self.glob    : B =/= A
  ------
  received:
  -----
  Super.GetProp: A  =  A
  Self.GetProp : B =/= A
  Self.prop    : B =/= A
  Self.prop2   : A  =  A
  ------
  
  -> FAILED

This is not NG itself being broken; the program in question prints the expected output when run normally in MaxIDE.